Predictive motion planning system and method

Abstract

A method and system for training a motion planner (330) for an autonomous vehicle (100) is described. A trajectory evaluator (334) agent of the motion planner (330) receives state data defining a current state of the autonomous vehicle (100) and an environment transition at a current time (802). Based on the current state, a trajectory is selected (804). A reward is calculated (806) based on the performance of the selected trajectory in the current state. State data (808) for a next state of the autonomous vehicle (100) and an environment transition at a next time is received. Parameters (810) of the trajectory evaluator (334) agent are updated based on the current state, the selected trajectory, the calculated reward, and the next state. The parameters of the trajectory evaluator (334) agent are updated to assign an evaluation value to the selected trajectory that reflects the calculated reward and expected performance of the selected trajectory in a future state.

Description

[0001] Cross-reference to related applications

[0002] This patent application claims the benefit of priority to U.S. patent application serial number 16 / 940,807, filed on July 28, 2020, entitled “PREDICTIVE MOTION PLANNING SYSTEM AND METHOD,” which is incorporated herein by reference as if reproduced in its entirety. Technical Field

[0003] The present disclosure relates to systems and methods for path planning for autonomous vehicles, including systems and methods for motion planning based on trajectory generation, evaluation, and selection. Background Art

[0004] Autonomous vehicles use various sensors to perceive their surroundings and the vehicle's own operating parameters. Autonomous vehicles typically include a planning system that plans future vehicle maneuvers based on endpoint goals and sensory data. The vehicle control system generates control commands based on the planning system's output. Therefore, the planning system plays a crucial role in decision-making, planning, and navigation.

[0005] The planning system can be decomposed into the following hierarchical planners (also known as planning units, planning subsystems, planning modules, etc.): a task planner, a behavior planner, and a motion planner. The motion planner receives the planned behaviors from the behavior planner and information about the perceived environment as input to output a trajectory for the autonomous vehicle. In this disclosure, a trajectory is a sequence of position coordinates and directions that the autonomous vehicle should follow over multiple time hops. Other parameters such as vehicle speed and / or vehicle acceleration can be defined in the trajectory.

[0006] Some methods for motion planning include trajectory generators, trajectory evaluators, and trajectory selectors. Trajectory generators typically use some prior knowledge (e.g., vehicle dynamics and kinematics) to generate only plausible trajectories and reduce the search space. Trajectory evaluators typically evaluate the generated trajectories (e.g., using some quantitative metric) to compare and rank them. Trajectory selectors typically select a desired trajectory based on the evaluation of the generated trajectories.

[0007] Existing rule-based motion planning techniques typically require optimization based on an explicitly defined cost function. Such rule-based approaches are often difficult to maintain and do not scale well to complex dynamic scenes. Furthermore, explicitly defined cost functions are often not general enough and difficult to adjust to account for complex dynamic interactions.

[0008] Other motion planning techniques use machine learning-based methods that learn from human driving patterns. For example, Zeng et al. (“End-to-End Interpretable Neural Motion Planner,” IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019) describe an end-to-end supervised learning approach that learns to assign better evaluations to trajectories that are more similar to human driving data and worse evaluations to trajectories that are more different from human driving data. In another example, Bansal et al. (“ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worst,” Robotics: Science and Systems, 2019) use a neural network to predict future trajectories directly from its history based on human driving data, and then use this predicted trajectory as the output of a motion planner.

[0009] These methods rely on human experts for training and are often limited to the experts' performance and experience. These methods may not fully explore parts of the state-action space that may be less important for safety and performance. Therefore, it would be beneficial to provide a machine learning-based motion planning method that does not rely on expert performance. Summary of the Invention

[0010] In various examples, this disclosure describes methods and systems for using machine learning-based methods for motion planning in autonomous vehicles. Various examples of the machine learning-based methods are described, including a method for training a trajectory estimator based on an RL agent that enables the trajectory estimator to learn from its prior experience rather than being limited to human-generated training data.

[0011] In various examples, the present disclosure describes a motion planner that receives as input a current state (which can include information about the perceived environment as well as information about planned behavior) and optionally a prediction in a future time jump, and selects a planned trajectory to follow based on optimization of a particular metric. Generally, a planned trajectory can be selected to achieve a safer and faster (but within the speed limit) driving experience.

[0012] In various examples, this disclosure describes an RL method for training a trajectory evaluator to assign evaluation values ​​to candidate trajectories. The trajectory evaluator can be trained using training data that includes explicitly defined trajectories (e.g., mathematically defined by an explicit function or formula), or it can be trained using training data that includes trajectory parameters. Training with explicitly defined trajectories allows for more flexibility in how trajectories are generated. Training with trajectory parameters allows for faster training and more thorough exploration of smaller trajectory spaces.

[0013] In some examples, the trajectory estimator can generate probability estimates characterized by statistical metrics. This is achieved through training using distributional reinforcement learning.

[0014] In some examples, the present disclosure describes a system for training a motion planner for an autonomous vehicle. The system includes a processing unit configured to execute instructions that cause the system to: receive a first set of state data as input to a trajectory evaluator agent of the motion planner, the first set of state data defining a current state of the autonomous vehicle and an environment transition at a current time; select a selected trajectory based on the current state; determine a calculated reward for the selected trajectory, the calculated reward being calculated based on the performance of the selected trajectory at the current state; receive a second set of state data defining a next state of the autonomous vehicle and an environment transition at a next time; and update parameters of the trajectory evaluator agent based on the current state, the selected trajectory, the calculated reward, and the next state, the parameters of the trajectory evaluator agent being updated to assign an evaluation value to the selected trajectory that reflects the calculated reward and expected performance of the selected trajectory in a future state.

[0015] In some examples, the present disclosure describes a method for training a motion planner for an autonomous vehicle. The method includes: receiving a first set of state data as input to a trajectory evaluator agent of the motion planner, the first set of state data defining a current state of the autonomous vehicle and an environment transition at a current time; selecting a selected trajectory based on the current state; determining a calculated reward for the selected trajectory, the calculated reward being calculated based on the performance of the selected trajectory at the current state; receiving a second set of state data defining a next state of the autonomous vehicle and an environment transition at a next time; and updating parameters of the trajectory evaluator agent based on the current state, the selected trajectory, the calculated reward, and the next state, the parameters of the trajectory evaluator agent being updated to assign an evaluation value to the selected trajectory that reflects the calculated reward and expected performance of the selected trajectory in a future state.

[0016] In some examples, the present disclosure describes a computer-readable medium storing instructions for execution by a processing unit of a system for training a motion planner for an autonomous vehicle, the instructions, when executed, causing the system to: receive a first set of state data as input to a trajectory evaluator agent of the motion planner, the first set of state data defining a current state of the autonomous vehicle and an environment transition at a current time; select a selected trajectory based on the current state; determine a calculated reward for the selected trajectory, the calculated reward being calculated based on the performance of the selected trajectory in the current state; receive a second set of state data defining a next state of the autonomous vehicle and an environment transition at a next time; and update parameters of the trajectory evaluator agent based on the current state, the selected trajectory, the calculated reward, and the next state, the parameters of the trajectory evaluator agent being updated to assign an evaluation value to the selected trajectory that reflects the calculated reward and expected performance of the selected trajectory in a future state.

[0017] In any of the above items, the first set of status data and the second set of status data may each independently include status data encoded in the form of a 2D image.

[0018] In any of the above items, the parameters of the trajectory estimator agent can be updated according to the following equation: V(s t ,τ t )←r t +γV(s t+1 ,τ t+1 ); where s t is the current state at the current time jump t, τ t is the selected trajectory, r t is the computational reward, V(s t ,τ t ) is the evaluation value of the selected trajectory at the current time jump and the current state, t+1 is the next time jump, V(s t+1 ,τ t+1 ) is the evaluation value of the selected trajectory at the next time jump and the next state, and γ is the discount factor.

[0019] In any of the above items, the parameters of the trajectory estimator agent can be updated according to the following equation: V(s t ,τ t )←r t +γV(s t+1 ,TS(s t+1 )); where s t is the current state at the current time jump t, τ t is the selected trajectory, rt is the computational reward, V(s t ,τ t ) is the evaluation value of the selected trajectory at the current time jump and the current state, t+1 is the next time jump, TS(s t+1 ) is the next selected trajectory to jump at the next time, V(s t+1 ,TS(s t+1 )) is the evaluation value of the next selected trajectory and the next state, and γ is the discount factor.

[0020] In any of the above, the estimate may be generated as a set of statistical measures defining a probability distribution of the probability estimate.

[0021] In any of the above, the selected trajectory may be selected according to a selection criterion based on one or more statistical metrics.

[0022] In any of the above items, the set of data defining the candidate trajectories may be a set of parameters defining the candidate trajectories according to a trajectory generation function.

[0023] In any of the above items, the instructions may further cause the system (or the method may include): generating the selected trajectory from the set of parameters according to the trajectory generation function.

[0024] In any of the above items, the selected trajectory may be defined by a set of 2D images, the set of 2D images defining waypoints of the candidate trajectory over a plurality of time hops. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Reference will now be made, by way of example, to the accompanying drawings which show exemplary embodiments of the present application, in which:

[0026] Figure 1 is a block diagram illustrating some components of an exemplary autonomous vehicle;

[0027] Figure 2 is a diagram showing a method for implementing Figure 1 A block diagram of some components of a processing unit of a path planning system for an autonomous vehicle is shown;

[0028] Figure 3 is a block diagram illustrating further details of an exemplary path planning system;

[0029] Figure 4 is a block diagram illustrating an exemplary training phase of a trajectory estimator in a motion planner;

[0030] Figure 5 shows examples of different state data encoded as 2D images;

[0031] Figure 6 An exemplary implementation of a motion planner is shown;

[0032] Figure 7 Another exemplary implementation of a motion planner is shown;

[0033] Figure 8 is a flow chart illustrating an exemplary method for training a motion planner.

[0034] Like reference numerals may be used in different drawings to identify like components. DETAILED DESCRIPTION

[0035] Some examples of the present disclosure are described in the context of self-driving cars. Although the examples described herein may refer to cars as self-driving cars, the teachings of the present disclosure can be implemented in other forms of autonomous or semi-autonomous vehicles, including, for example, trams, subways, trucks, buses, surface and underwater vehicles and ships, airplanes, drones (also known as unmanned aerial vehicles (UAVs)), warehouse equipment, manufacturing facility equipment, construction equipment, agricultural equipment, autonomous service robots such as vacuum cleaners and lawn mowers, and other robotic devices. Self-driving cars may include vehicles without passengers as well as vehicles with passengers.

[0036] Figure 1 is a block diagram illustrating certain components of an exemplary autonomous vehicle 100. Although described as autonomous, vehicle 100 can operate in fully autonomous, semi-autonomous, or fully user-controlled modes. In this disclosure, vehicle 100 is described in the context of an automobile; however, as noted above, the disclosure can be implemented in other autonomous vehicles.

[0037] Automobile 100 includes, for example, a sensor system 110, a perception system 120, a path planning system 130, a vehicle control system 140, and an electromechanical system 150. Other systems and components may be included in automobile 100 as appropriate. In this example, perception system 120, path planning system 130, and vehicle control system 140 are distinct software systems comprising machine-readable instructions executable by one or more processors in a processing unit of automobile 100. Alternatively, perception system 120, path planning system 130, and vehicle control system 140 may be distinct systems implemented on one or more chips (e.g., application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), and / or other types of chips). For example, perception system 120, path planning system 130, and vehicle control system 140 may be implemented using one chip, two chips, or three different chips (using the same or different types of chips). The various systems and components of the vehicle may communicate with each other, for example, via wired or wireless communications. For example, the sensor system 110 can communicate with the perception system 120 and the vehicle control system 140; the perception system 120 can communicate with the path planning system 130 and the vehicle control system 140; the path planning system 130 can communicate with the vehicle control system 140; and the vehicle control system 140 can communicate with the electromechanical system 150.

[0038] Figure 1 An example data flow (indicated by arrows) is shown from sensor system 110 to perception system 120, path planning system 130, vehicle control system 140, and mechatronic system 150 (discussed in further detail below). However, it should be understood that data communication between systems 110, 120, 130, 140, 150 may be performed in a variety of different ways, and that bidirectional data communication may exist between systems 110, 120, 130, 140, 150.

[0039] The sensor system 110 includes various sensing units, such as a radar unit 112, a lidar unit 114, and a camera 116, for collecting information about the environment surrounding the car 100 as the car 100 operates in the environment. The sensor system 110 also includes a global positioning system (GPS) unit 118 for collecting information about the vehicle's position in the environment. The sensor system 110 also includes one or more internal sensors 119 for collecting information about the physical operating conditions of the car 100 itself, including, for example, sensors for sensing steering angle, linear velocity, linear and angular acceleration, attitude (pitch, yaw, roll), compass heading, vehicle vibration, throttle status, brake status, wheel traction, transmission gear ratio, cabin temperature and pressure, etc. The sensor system 110 may include Figure 1 Other sensors not shown include, for example, environmental sensors for sensing external conditions such as external temperature and pressure, precipitation, and noise.

[0040] The information collected by the sensor system 110 is provided to the perception system 120 as sensor data. The perception system 120 processes the sensor data to generate state data regarding the internal state of the vehicle and state data regarding the state of the environment surrounding the vehicle 100. The state data regarding the internal state of the vehicle may include, for example, data defining the vehicle's position (e.g., including data regarding the position and orientation of the vehicle 100, which may be absolute geographic longitude / latitude values ​​and / or values ​​referenced to other reference frames) and / or data defining the operation of the vehicle (e.g., including data regarding the vehicle's kinematic parameters, such as linear speed and acceleration, direction of travel, angular acceleration, attitude (e.g., pitch, yaw, roll) and vibration, as well as mechanical system operating parameters, such as engine RPM, throttle position, brake position, and transmission gear ratio). The state data regarding the state of the environment surrounding the vehicle 100 may include, for example, information regarding stationary and moving objects detected around the vehicle 100, weather and temperature conditions, road conditions, road configuration, and other information regarding the physical environment.

[0041] Generally speaking, the term kinematics refers to a class of problems in robotics and motion planning where velocity, acceleration, and force / torque bounds must be met, as well as kinematic constraints (e.g., obstacle avoidance). Kinematics parameters are parameters associated with such motion planning problems, as described above. In some embodiments, kinematics parameters may include one or more of the following: maximum acceleration, maximum deceleration, maximum steering wheel speed of the vehicle 100, a kinematic model of the vehicle 100, the width or length of the vehicle 100, the minimum turning radius of the vehicle, etc.

[0042] In some examples, sensor system 110 or perception system 120 may also access a map (not shown), which may be a reference map representing known information about the surrounding environment. For example, the map may be received from an external reference database (e.g., retrieved based on the vehicle's location determined using GPS unit 118).

[0043] For example, sensor data received from radar 112, lidar 114, and camera unit 116 can be used to determine the local environment of car 100 (e.g., any surrounding obstacles) as well as information from the wider vicinity (e.g., radar unit 112 and lidar unit 114 can collect information from an area with a radius of up to 100 meters or more around car 100). Sensor data from GPS unit 118 and other sensors can be used to determine the location of the vehicle, thereby defining the geographic location of car 100. Sensor data from interior sensors 119 and from other sensor units can be used to determine vehicle kinematic parameters, including the velocity and attitude of car 100 relative to a reference frame.

[0044] The perception system 120 in this disclosure is described as a general-purpose system that performs various sensing and processing functions. In some implementations, the perception system 120 can be implemented using various subsystems for performing different functions. For example, the perception system 120 can provide sensing, sensor fusion, localization, semantic understanding, object detection and tracking, and / or processing world models. In some actual implementations, some of these functions can be implemented using different subsystems within the perception system 120 (e.g., a fusion subsystem or a state estimator subsystem). For simplicity and without loss of generality, this disclosure generally refers to the perception system 120. The data output by the perception system 120 can include data that has been minimally processed (e.g., normalized and / or filtered to remove noise). The perception system 120 can also perform more extensive processing on the sensor data, such as generating an occupancy grid map (OGM) and predicted future OGMs, detecting and classifying objects of interest (e.g., other vehicles, pedestrians, etc.), detecting the position and velocity of objects classified by their class, detecting road lane markings and the location of lane centers, and so on. Thus, the data output by the perception system 120 may include analytical (or estimated) data (eg, OGM and object classification) as well as simple data (eg, vehicle speed, vehicle acceleration, etc.).

[0045] Data is output in real time from perception system 120 to planning system 130, which is the focus of this disclosure and will be described in more detail below. Vehicle control system 140 is used to control the operation of automobile 100 based on the endpoint goals set by planning system 130. Vehicle control system 140 can be used to provide full, partial, or auxiliary control of automobile 100. Electromechanical system 150 receives control signals from vehicle control system 140 for operating electromechanical components of automobile 100, such as the engine, transmission, steering system, and braking system.

[0046] As previously mentioned, perception system 120 , planning system 130 , and vehicle control system 140 may be implemented as software comprising machine-readable instructions executable by one or more processing units of a processing system of automobile 100 .

[0047] Figure 2 An example of a processing unit 200 that can be implemented in the automobile 100 is shown. The processing unit 200 includes one or more processors 210. The one or more processors 210 may include a central processing unit (CPU), a graphical processing unit (GPU), a tensor processing unit (TPU), a neural processing unit (NPU), a digital signal processor, or other computing elements. The processor 210 is coupled to an electronic memory 220 and one or more input and output (I / O) interfaces or devices 230. The electronic memory 220 may include one or more non-volatile or tangible memories (e.g., flash memory) and / or volatile memories (e.g., RAM). The tangible memory may store instructions, data, and / or software modules for execution by the processor 210 to perform the functions of the system described herein. The tangible memory of the electronic memory 220 may store other software instructions and data used to implement other operations of the automobile 100.

[0048] Electronic storage 220 may include any suitable volatile and / or non-volatile storage and retrieval means, including, for example, flash memory, random access memory (RAM), read-only memory (ROM), hard disks, optical disks, subscriber identity module (SIM) cards, memory sticks, secure digital (SD) memory cards, and other state storage devices. Figure 2In the example shown in FIG2 , electronic storage 220 of processing unit 200 stores instructions (executable by processor 210) for implementing perception system 120, planning system 130, and vehicle control system 140. In some embodiments, electronic storage 220 also stores data 145, including sensor data provided by sensor system 110 and data used by planning system 130 to generate a driving plan, such as a map of the current environment.

[0049] Figure 3 is a block diagram showing further details of planning system 130; for clarity, some components of car 100 are omitted. It should be understood that in some examples involving machine learning, planning system 130 or modules or subsystems of planning system 130 can be trained outside of car 100 (e.g., in a simulator system). In the examples discussed below, at least motion planner 330 can be trained in a simulator.

[0050] In general, planning and decision making can be performed at three levels: at the task level (e.g., performed by task planner 310), at the behavior level (e.g., performed by behavior planner 320), and at the motion level (e.g., performed by motion planner 330).

[0051] Typically, the purpose of planning and decision-making is to determine a path for the vehicle 100 to travel from a first location (e.g., the vehicle's current location and orientation, or an expected future location and orientation) to a target location (e.g., a user-defined final destination). Planning may also include determining one or more sub-paths to one or more intermediate target locations. The planning system 130 determines appropriate paths and sub-paths taking into account conditions such as the drivable surface (e.g., defined roads), obstacles (e.g., pedestrians and other vehicles), traffic rules (e.g., obeying traffic signals), and user-defined preferences (e.g., avoiding toll roads).

[0052] The planning and decision making of the planning system 130 can be dynamic and repeatedly executed as the environment changes. The changes in the environment can be due to the movement of the car 100 (e.g., the car 100 approaches a newly detected obstacle) and due to the dynamic properties of the environment (e.g., moving pedestrians and other moving vehicles).

[0053] As mentioned above, planning and decision-making can occur at different levels, such as the task level, the behavior level, and the movement level. Task-level planning is considered a higher (or more global) level of planning, movement-level planning is considered a lower (or more local) level of planning, and behavior-level planning is considered to be somewhere between the task level and the movement level. Generally speaking, the outputs of higher-level planning and decision-making can form at least part of the inputs to lower-level planning and decision-making.

[0054] Mission-level planning (more simply referred to as mission planning) involves planning a path for the autonomous vehicle at a higher or global level. The first location of the car 100 can be the starting point of the journey (e.g., the user's home), and the target location of the car 100 can be the final destination point (e.g., the user's workplace). Selecting a route through a set of roads is an example of mission planning. Typically, the final destination point, once set (e.g., by user input), remains unchanged for the duration of the journey. While the final destination point may not change, the path planned by the mission plan may change over the duration of the journey. For example, changing traffic conditions may require the mission plan to dynamically update the planned path to avoid congested roads. The user can also change the final destination point at any time during the journey.

[0055] Input data for mission planning may include, for example, GPS data (e.g., determining a starting point for the car 100), geographic map data (e.g., from an internal or external map database), traffic data (e.g., from an external traffic condition monitoring system), a final destination point (e.g., defined as x and y coordinates, or as latitude and longitude coordinates), and any user-defined preferences (e.g., a preference for avoiding toll roads).

[0056] The planned path output from the mission planner 310 defines the route to be taken from the starting point to the final destination point. The planned path output from the mission planner 310 may include a set of intermediate target locations (or waypoints) along the route. For example, intermediate target locations may be defined at road intersections to indicate which road to take at each intersection. The intermediate target locations may be used for behavioral-level planning (e.g., decision making to determine appropriate vehicle behavior).

[0057] The behavior planner 320 receives the planned path output from the mission planner 310, including the set of intermediate target locations (if any). The behavior planner 320 generates behavioral decisions to control the behavior of the vehicle 100 on a more local and short-term basis than the mission planner 310. The behavior planner 320 also generates output representing the behavioral decisions, which can be used as targets for the motion planner 330. The behavior planner 320 can generate behavioral decisions based on certain rules or guidelines. For example, the behavior planner 320 can ensure that the vehicle 100 follows certain behavioral rules (e.g., turning left from the leftmost lane, driving within the speed limit, stopping at stop signs, etc.). Such behavioral rules can be based on traffic regulations or on guidance for smooth and efficient driving (e.g., the vehicle should use the faster lane when possible). The output representing the behavioral decisions generated by the behavior planner 320 can be in various suitable formats. For example, the output representing the behavioral decisions generated by the behavior planner 320 can be in the form of commands, signals (e.g., safety signals), scalar values, and / or cost maps (or cost images). The output representing the behavior decisions generated by the behavior planner 320 can be used as constraints for motion planning, for example.

[0058] The output representing the behavioral decision can be in the form of a command provided to the motion planner 330. For example, the command provided to the motion planner 330 can include motion planning constraints. The motion planner 330 should find a trajectory that satisfies the behavioral decision and navigates the environment in a relatively safe, comfortable and fast manner. The motion planner 330 should be designed to provide safe and reliable navigation in structured and unstructured environments. A structured environment is typically an environment with clearly defined drivable and non-drivable areas (e.g., a highway with clear lane markings), and may define driving rules that all vehicles should follow. An unstructured environment typically refers to an environment with fewer (or undefined) drivable and non-drivable areas (e.g., open land) and may have fewer or no driving rules for expected vehicle behavior. Regardless of whether the environment is structured or unstructured, the environment may be highly dynamic (e.g., pedestrians and other vehicles are all moving), and each dynamic obstacle may have different and independent behaviors. This makes the design of the motion planner 330 challenging.

[0059] exist Figure 3In the example shown, the motion planner 330 includes a state encoder 331, a trajectory generator 332, a trajectory evaluator 334, and a trajectory selector 336. The state encoder 331 generates the current state of the car 100 using state data about the internal state of the vehicle, state data about the surrounding environment of the car 100 received from the perception system 120, and output representing the behavior decision generated by the behavior planner 320. The state of the car 100 is defined by the sensed environment around the vehicle (e.g., detected obstacles, detected lane markings, etc.) and the sensed environment inside the vehicle (e.g., the vehicle's speed, acceleration, etc.) sensed by the perception system 120. The state s of the car 100 t The behavior decision definitions are also generated by the behavior planner 320, as will be discussed further below. The trajectory generator 332 can be implemented using various suitable methods, such as using expert-designed equations (e.g., polynomials) to generate a trajectory for the current state of the car 100. The trajectory evaluator 334 receives the current state of the car 100 and at least one candidate trajectory as input from the trajectory generator 332 and assigns an evaluation value to each candidate trajectory. The assigned evaluation value reflects whether the candidate trajectory successfully achieves the goals of relatively safe, comfortable, and fast driving (also satisfying the behavior decision). The trajectory evaluator 334 in the present disclosure is implemented using machine learning. In the example discussed below, the trajectory evaluator 334 is a software agent (hereinafter referred to as the agent) that is trained to learn to correctly assign values ​​to the functions of the candidate trajectories. The training of the trajectory evaluator 334 is based on reinforcement learning (RL), performed through interaction with the environment (real or simulated), and receiving reinforcement rewards that quantify the direct impact of the trajectory followed by the real or simulated car 100. In some embodiments, the agent is implemented using a neural network that approximates the function that correctly assigns values ​​to the candidate trajectories. The parameters of the neural network are learned based on RL. The trajectory selector 336 selects the candidate trajectory with the highest evaluation value (assigned by the trajectory evaluator 334) from among the candidate trajectories generated by the trajectory generator 332.

[0060] Although the state encoder 331, trajectory generator 332, trajectory evaluator 334, and trajectory selector 336 are shown and discussed herein as separate blocks (e.g., separate modules or separate functions) implemented in the motion planner 330, it should be understood that two or more of the state encoder 331, trajectory generator 332, trajectory evaluator 334, and trajectory selector 336 can be implemented together in a single block (e.g., in the same module or algorithm).

[0061] Figure 4is a block diagram illustrating details of an example motion planner 330 in the context of training a trajectory estimator 334 of the motion planner 330. After sufficient training (e.g., the learning function and / or parameters (e.g., weights) of the trajectory estimator 334 have converged), the trained trajectory estimator 334 of the motion planner 330 can be used for inference and can be implemented in the autonomous vehicle 100. For example, the trained motion planner 330 can be used in a Figure 3 1 is deployed (eg, implemented) in the planning system 130. When the trained trajectory estimator 334 is deployed for inference, the reward processor 410 and the training processor 412 (discussed further below) are disabled.

[0062] The motion planner 330 includes a state encoder 331 that generates a current state (denoted as s ) based on state data regarding the internal state of the vehicle, state data regarding the state of the environment surrounding the car 100 received from the perception system 120, and output representing a behavior decision received from the behavior planner 320. t ). Can be used to generate the current state s t Some exemplary state data about the internal state of the vehicle and state data about the surrounding environment of the car 100 are shown in FIG. Figure 5 and discussed further below.

[0063] In some examples, training of motion planner 330 can be continuous and ongoing, and can enable motion planner 330 to be further refined based on actual operation of autonomous vehicle 100.

[0064] In the example shown, the training process involves an RL process to enable the trajectory evaluator 334 to learn to generate appropriate estimated evaluation values ​​and assign estimated evaluation values ​​to candidate trajectories. In machine learning-based approaches, particularly RL-based approaches, as discussed herein, the trajectory evaluator 334 can be an agent implemented using a neural network. The agent of the trajectory evaluator 334 can be implemented using any suitable neural network architecture, including a convolutional neural network (CNN), a recurrent neural network (RNN), or a deep neural network, among others.

[0065] A general discussion of the training of the trajectory evaluator 334 is now presented. In the context of the present disclosure, a trajectory is a sequence of waypoints that defines a planned path for an autonomous vehicle over a specific time period over multiple time hops. A trajectory can include waypoints that define, for each time hop, a planned vehicle position, a planned vehicle heading, a planned vehicle speed, and / or a planned vehicle acceleration, among other possible parameters. In some examples, a trajectory can be represented as a vector with a sequence of waypoints as entries in the vector. If each waypoint contains multiple parameters (e.g., a waypoint defines a planned vehicle position and heading), then the trajectory can be represented as a multi-dimensional vector.

[0066] Jump at the current time t and provide candidate trajectory τ t As the input of the trajectory evaluator 334. Candidate trajectory τ t The trajectory generator 332 can be based on the current state s t Generate candidate trajectories τ t The trajectory estimator 334 also receives the current state s t As input. Current state s t The state of the external environment around the car 100, the internal state of the car 100, and the behavior decision generated by the behavior planner 320 can be represented by the trajectory evaluator 334. t and candidate trajectory τ t Mapped to the evaluation value V(s t ,τ t ). This evaluation value is used to evaluate the suitability of the candidate trajectory for use by the autonomous vehicle 100. As described above, the trajectory evaluator 334 can be an agent, and the agent can be implemented using a neural network. During training, the parameters of the neural network can be learned. The RL training processor 412 runs (e.g., executes) a reinforcement learning algorithm to update the parameters (e.g., weights) of the neural network until a termination criterion is met (e.g., performance has reached a minimum threshold).

[0067] The trajectory evaluator 334 estimates the candidate trajectory τ t Evaluation value V(s t ,τ t ). A candidate trajectory (also called a selected trajectory) may be selected, and the autonomous vehicle 100 will follow the trajectory. When the autonomous vehicle 100 follows the selected trajectory, the autonomous vehicle 100 and the sensed environment change from the current state s t Transition to the next state s t+1If the training process is performed in a simulator (e.g., for safety and efficiency purposes), components of the autonomous vehicle 100, such as the vehicle control system 140, the electromechanical system 150, the sensor system 110, and the perception system 120, may be simulated in the simulator, as may the environment. t The perceived impact of the environment (whether simulated or real) generates feedback to the trajectory evaluator 334. In RL, feedback can take the form of a reward r t , which may be generated by the reward processor 410. The reward processor 410 may be implemented using a dedicated hardware device (e.g., having software instructions implementing a reward algorithm), or may be implemented using a software model executed by the processing unit 200 of the car 100, or a combination thereof. Similarly, the RL training processor 412 may be implemented as hardware, software, or a combination thereof. If implemented as hardware, the reward processor 410 and the RL training processor 412 may be implemented together or separately in dedicated hardware. The reward processor 410 may receive the current state s t , selected action (i.e., selected trajectory τ t ) and the next state s t-1 , and output reward r t Alternatively, the reward processor 410 may receive only the next state s t-1 And output reward r t . Reward r t reflects the performance in the environment of the selected trajectory. The reward processor 410 is typically designed so that the generated reward r t Encourage safety, comfort, and mobility of the autonomous vehicle 100. Typically, when a trajectory τ is selected t The reward processor 410 generates a higher (or corrected) reward r as the desired goal of a safe, comfortable, and fast path is approached. t For example, if the selected trajectory τ t The autonomous vehicle 100 drives along the lane at a speed close to the speed limit, and the reward r t is positive. On the contrary, if the selected trajectory τ t If the self-driving car 100 gets into an accident, the reward is r t is negative. By interacting and observing different states s t Different trajectories τ t Reward r t , the trajectory evaluator 334 is trained to assign estimated evaluation values ​​that more accurately reflect the desired target.

[0068] Reward t The trajectory evaluator 334 is used to learn the parameters (e.g., weights) of a neural network (discussed further below) for generating an estimated evaluation value V(s) using RL. t,τ t As shown, the trajectory estimator 334 can be trained using an RL training processor 412 that executes an RL algorithm based on Q-learning as follows:

[0069] V(s t ,τ t )←(r t +γV(s t+1 ,τ t+1 ))(1)

[0070] where s t is the state at time t, s t+1 is the next state at time t+1, τ t is the selected trajectory being evaluated at time t, τ t+1 is the next selected trajectory at time t+1, r t is the reward received at time t, V(s t ,τ t ) is for state s t and trajectory τ t The estimated evaluation value. In addition, 0<γ<1 is the discount factor, which is a scalar value that discounts the value of the future state. The symbol ← represents the value of the future state based on (r t +γV(s t+1 ,τ t+1 ))Update V(s) t ,τ t ), but it should be noted that this update is not necessarily a direct update, but rather an indication of the Figure 4 The other blocks shown are from (r t +γV(s t+1 ,τ t+1 )) to V(s t ,τ t During the inference phase, the reward processor 410 and the RL training processor 412 are disabled.

[0071] Notably, because trajectory evaluator 334 is not trained using human-generated driving data, trajectory evaluator 334 can be trained to better evaluate a variety of different trajectories (including trajectories that may be less likely to be explored in human-generated driving data) under various complex conditions (including conditions that may be dangerous or difficult for humans to drive).

[0072] As described above, trajectory evaluator 334 may be an agent implemented using a neural network. In some examples, the state s input to trajectory evaluator 334 is tIt may be in the form of a 2D image. For example, state data representing spatial information (eg, OGM, objects, lane markings, etc.) may be in the form of a 2D image.

[0073] Figure 5 shows how state data can be encoded by state encoder 331 into state s t Some examples of this. In general, a 2D image can include one or more channels. In a color 2D image, there are typically three channels encoding red, green, and blue intensities. Adapting this concept to state data, different channels can be used to encode different forms of information in the form of a 2D image, as described below. State data in the form of 2D images can be concatenated together (e.g., as multiple channels of a single 2D image) to form a state s t .

[0074] For example, the location of each detected object can be encoded as a 2D box (e.g., corresponding to a bounding box generated by an object detection algorithm). Each object category (e.g., bicycle, car, truck, pedestrian, etc.) can be encoded in a 2D image using a corresponding separate channel, or can be represented by a corresponding different color in a single 2D image. In the example shown, the state data includes lane markings 421 (or other map information), which can be encoded in a single 2D image with separate channels to encode the left, right, and center of the lane. Lane markings and / or drivable areas can be encoded in the 2D image as three lines, for example, representing the left boundary, right boundary, and centerline of the lane. Each line can be encoded using a corresponding separate channel of the image, or can be represented in a single 2D image using a corresponding different color (e.g., black for the left boundary, green for the centerline, and brown for the right boundary). Object speed 422 can be encoded in the 2D image as an intensity signal. For example, the speed of each object can be encoded as a line proportional to the speed value and along its direction of movement. The line can be further encoded as having a higher intensity for the starting position (e.g., at the current time) and a lower intensity as the line extends toward the ending position (e.g., at the projected position of the next time jump). Object categories 423 can be encoded using different channels for each object category of interest (e.g., car, bicycle, pedestrian, etc.). The behavior decision 424 from the behavior planner 320 can be encoded as an intensity signal indicating the target behavior of the car 100 (e.g., continue straight). Each OGM frame 425 can be encoded in the form of a 2D or 3D matrix. For example, a 3D matrix can be encoded as a 2D image with multiple channels. In the case where the OGM frame 425 is a 3D matrix, the height or depth of the image can contain information indicating the occupancy of objects at different heights or depths, respectively. In some examples, the OGM frame 425 can be represented by two 2D images, where one 2D image indicates occupancy at a predefined height and the second 2D image indicates changes in occupancy with the predefined height. In addition, each channel of the OGM frame 425 can correspond to a different corresponding time stamp, for example, from a predefined past time point (e.g., a predefined number of time jumps from the current time) to a predefined point in the future (e.g., a prediction of a predefined number of future time jumps). The OGM frame 425 includes at least one channel corresponding to the current time stamp (i.e., corresponding to the current sensor data). Scalar information (e.g., vehicle internal state 426 including values ​​representing the vehicle internal state, such as speed, acceleration, etc.) can also be encoded as a 2D image in a separate channel. Alternatively, the scalar information can be fed directly to the trajectory estimator 334 at the fully connected layer of the neural network.

[0075] It should be noted that the behavioral decision 424 may also be input to the trajectory generator 332 (whether encoded as a 2D signal or not as described above) to influence the generation of the trajectory. For a given behavioral decision, certain trajectories may be relevant, while other trajectories may not be allowed. For example, if the behavioral decision is to turn right, a curved trajectory may be relevant, but a straight trajectory may not be appropriate. In another example, if the behavioral decision is to change lanes to the right, relevant trajectories may include a straight trajectory (e.g., to stay in the current lane in the event that a lane change cannot be safely executed) and a right-turn trajectory, but a left-turn trajectory may not be appropriate. In another example, if the behavioral decision is to stop at a stop sign, the trajectory generator may limit the target speed to force a stop before the stop sign.

[0076] Other techniques for encoding data into 2D images can be used. For example, instead of using separate channels to encode different object categories, all object categories of interest can be encoded into a single channel of the 2D image by assigning different intensity ranges to each object category (e.g., pixel intensities of 0-19 represent cars, pixel intensities of 20-39 represent bicycles, etc.). Additional information (e.g., weather, object acceleration, predicted object behavior, etc.) can also be encoded into the 2D image in a similar manner.

[0077] Encoding state data into 2D images can facilitate the use of CNNs (which are well-studied for processing 2D images) to implement trajectory estimator 334. Although exemplary embodiments using CNNs to implement trajectory estimator 334 are discussed, it should be understood that trajectory estimator 334 can be an agent implemented using any suitable neural network (or suitable machine learning technique). For example, the agent of trajectory estimator 334 can be implemented using an RNN.

[0078] Figure 6 is a block diagram illustrating an example implementation of the motion planner 330 , showing the training of the trajectory estimator 334 .

[0079] In some examples, trajectory generator 332 can be implemented using parameter generator 337 and trajectory sub-generator 338. Parameter generator 337 generates parameters (e.g., desired velocity, lateral offset, etc.) to define candidate trajectories to be generated. During training, parameter generator 337 can randomly select parameter values ​​from a defined range (e.g., based on input from behavioral decision making). Trajectory sub-generator 338 uses the trajectory parameters to generate candidate trajectories τ. t .

[0080] In this example, each candidate trajectory τ generated by the trajectory sub-generator 338 tIn the form of a 2D image, a trajectory is explicitly defined as a set of waypoints in the 2D image. The trajectory can be encoded in a 2D image in various ways. For example, a sequence of points connected by a line in the 2D image can represent the geographical 2D position of each waypoint at each time jump. The intensity (or thickness) of the line in the 2D image can encode the planned vehicle speed along the trajectory. Figure 6 As shown, the trajectory can be encoded as a 2D image with N channels corresponding to N future time jumps. In examples where the predicted OGM is part of the state data, the N future time jumps can correspond to the predicted level of the OGM received in the state data. For each channel, a box representing the autonomous vehicle 100 can be placed at the planned position of the autonomous vehicle 100 along the trajectory at the corresponding time jump.

[0081] The training processor 412 updates the parameters (e.g., weights) of the neural network of the trajectory estimator 334 using the following formula:

[0082] V(s t ,τ t )←r t +γV(s t+1 ,τ t+1 )(2)

[0083] For example, the trajectory evaluator 334 is learned by t , discounted estimated future valuation γV(s t+1 ,τ t+1 ) and the current estimated evaluation value update function V(s t ,τ t ) to execute. An example is:

[0084] δ=r t +γV(s t+1 ,τ t+1 )-V θ (s t ,τ t )

[0085]

[0086] where θ is the set of neural network weights, V θ is a neural network using weights θ, δ is the error (which will be returned to zero), α is the learning rate, and Based on the gradient of θ. The above equations illustrate an exemplary method for training the neural network weights of trajectory evaluator 334 using gradient descent. Other training techniques may be used.

[0087] It should be appreciated that equation (2) is based on equation (1) discussed above. The goal of equation (2) is to minimize the error between the evaluation value generated by trajectory evaluator 334 and the actual reward expected to be received. In other words, the training of trajectory evaluator 334 is to learn a value function that will satisfy equation (2). In this example, assume that the trajectory τ at time jump t+1 t+1 is the same trajectory τ with time jump t t The continuation of t is mapped to the time jump t+1. In other words, the estimated evaluation value V(s t ,τ t ) corresponds to the hypothetical situation where the autonomous vehicle 100, in addition to the time jump t, will also jump along the candidate trajectory τ at a future time t In this sense, it can be argued that although rewards are calculated based on short-term performance (e.g., instantaneous reward r t is calculated based on the performance of the next time jump), but the evaluation value corresponds to the expected long-term performance. Note that the autonomous vehicle 100 can actually jump along the trajectory τ at time exceeding t+1. t Therefore, the trajectory estimator 334 is trained to learn the value V(s t ,τ t ), without requiring the autonomous vehicle 100 to follow a trajectory τ exceeding t+1 t .

[0088] During the training phase, the autonomous vehicle 100 will follow the candidate trajectory τ at time jump t t As a time jump progresses, the autonomous vehicle 100 and the environment will transition to state s at time t+1. t+1 , which also generates a reward r in the process t At a given time jump t+1, the trajectory evaluator 334 is fed with a candidate trajectory τ t , current given state s t , next state s t+1 and reward r t In other words, the training data samples of the trajectory evaluator 334 are composed of (s t ,τ t ,r t ,s t+1 ), the training data set of the trajectory evaluator 334 is a certain range of i=0 to k (s t ,τ t ,r t ,s t+1 ) i The trajectory evaluator 334 is based on τ t and s t+1 Assume (as above) that τ t+1, so τ t+1 is τ t During the training phase, the trajectory selector 336 may select the trajectory τ with the highest evaluation value (evaluated by the trajectory evaluator 334) from the possible trajectories generated by the trajectory generator 332. t Trajectory selector 336 may also select a random trajectory among the possible trajectories generated by trajectory generator 332. The process of selecting a random trajectory is designed to improve the training of trajectory evaluator 334 by collecting data on trajectories that may be better but have not yet been tried (and evaluated) and / or are not included in the training dataset.

[0089] The trained trajectory evaluator 334 can then be implemented in an actual autonomous vehicle 100 for practical use. During inference, the reward processor 410 and the RL training processor 412 are disabled. During inference, the trajectory generator 332 generates (i.e., infers) multiple candidate trajectories for a given state at a given time. The trained trajectory evaluator 334 assigns a corresponding evaluation value to each of the multiple candidate trajectories for the given state at a given time. The trajectory selector 336 can then select a selected trajectory with the highest assigned evaluation value from all candidate trajectories. The selected trajectory is then provided to the vehicle control system 140 to generate control signals for the autonomous vehicle 100.

[0090] In the examples discussed above, trajectory evaluator 334 does not have any assumptions or prior knowledge about the candidate trajectories or about how trajectory generator 332 is designed. Therefore, trajectory evaluator 334 is trained to learn to correctly evaluate candidate trajectories regardless of how trajectory generator 332 is implemented. Therefore, trajectory generator 332 can be designed to generate candidate trajectories using any desired function (e.g., any polynomial) and / or based on any source (e.g., historical trajectories or human driving data). Furthermore, trajectory generator 332 can be dynamically adjusted to generate candidate trajectories for different tasks / conditions (e.g., based on behavioral decisions from behavior planner 320) without the need to retrain trajectory evaluator 334.

[0091] The examples above describe a training process that is independent of how candidate trajectories are generated. In many practical implementations, trajectories are typically generated using a polynomial function that defines candidate trajectories based on a finite set of scalar parameters. For example, the set of parameters defining a trajectory might include initial velocity, final velocity, initial direction, initial offset from lane center, final offset from lane center, and trajectory level.

[0092] Figure 7 is a block diagram illustrating another exemplary implementation of the motion planner 330 , showing the training phase of the trajectory estimator 334 .

[0093] In this example, the parameter generator 337 is separated from the trajectory generator 332. Instead of performing the evaluation on the candidate trajectory itself, the trajectory evaluator 334 is trained to evaluate the current state s t and will be used to generate the trajectory τ t The trajectory parameters are mapped to the evaluation value V(s t ,τ t Because the trajectory evaluator 334 is trained to estimate parameters to be used by the trajectory generator 332, the training of the trajectory evaluator 334 inherently takes into account how the trajectory is generated from the trajectory parameters (e.g., the type of polynomial used), and the trained trajectory evaluator 334 is specific to the particular design of the trajectory generator 332 (e.g., the particular type of polynomial used by the trajectory generator 332 to generate the trajectory).

[0094] During the training phase, the parameter generator 337 generates a set of parameters p t , this set of parameters is used to generate the trajectory. Parameter p t Instead of defining candidate trajectories explicitly, candidate trajectories are defined indirectly (or implicitly). Trajectory evaluator 334 is trained to evaluate parameters that trajectory generator 332 will use to generate trajectories, rather than directly evaluating candidate trajectories. Trajectory evaluator 334 is trained based on the effects of generated trajectories on the environment, even though trajectory evaluator 334 does not receive generated trajectories as input. In some examples, it may be assumed that the trajectory τ at the future time jump t+1 t+1 is used to generate the trajectory τ at time jump t t In other words, the autonomous vehicle 100 will jump along the candidate trajectory τ in the future time. t Assume that the trajectory generator 332 is trained to output the evaluation value V(s t ,τ t ).

[0095] In this example, the track selector 336 is used to execute the t to the parameter p that produces the highest evaluation value t The mapping function.

[0096] During the training phase, the trajectory estimator 334 is fed with the data at a given time jump t and the data at a given state s t The parameter set p corresponding to the trajectory followed at time t t , along the line corresponding to p t The next state s generated by the trajectory t+1 and the reward r t In other words, the training data samples of the trajectory evaluator 334 are composed of (s t ,p t ,r t ,s t+1), the training data set of the trajectory evaluator 334 is a certain range of i=0 to k (s t ,p t ,r t ,s t+1 ) i of the collection.

[0097] The trained trajectory estimator 334 can then be implemented in an actual autonomous vehicle 100 for practical use. During inference, the reward processor 410 and the RL training processor 412 can be disabled. During inference, the parameter generator 332 generates multiple parameter sets for a given state at a given time. The trained trajectory estimator 334 assigns a corresponding estimated evaluation value to each of the multiple parameter sets for the given state at a given time. The trained trajectory selector 336 can then select a selected parameter set for use by the trajectory generator 332 based on the given state. The trajectory generator 332 then generates a trajectory based on the selected parameter set and then provides the trajectory generated based on the selected parameter set (also referred to as the selected trajectory) to the vehicle control system 140 to generate control signals for the autonomous vehicle 100. In general, the trajectory selector 336 can be any suitable software module that finds a trajectory that maximizes the output of the trajectory estimator 334 in a given state. The trajectory selector 336 can be implemented in various ways, including the method described above. Other techniques for implementing trajectory selector 336 include, for example, trajectory selector 336 performing an iterative search, or a function that finds a trajectory that maximizes the output of trajectory evaluator 334 (mapping from a given state to a desired trajectory).

[0098] and Figure 6 Compared to the example in Figure 7 In the example of , training of trajectory evaluator 334 can be faster because the space of different trajectories to be evaluated is smaller (i.e., limited to the specific functions used by trajectory generator 332). In addition, performing training using trajectory parameters enables trajectory evaluator 334 to be trained on a continuous space of possible trajectories, rather than a finite number of explicit trajectories (which may not fully explore the entire trajectory space, or may explore the space in a piecemeal manner). Evaluation is performed in a continuous space, unlike Figure 6 Evaluating on a discrete (and finite) set of candidate trajectories may yield better performance than Figure 6 Compared with the example of Figure 7 Training the trajectory evaluator 334 in the manner shown enables faster training and better generalization (for a given trajectory generation function). In addition, the trajectory selector 336 can directly output the selected parameters without having to consider multiple candidate trajectories, thereby enabling faster processing during actual deployment in the inference phase.

[0099] In some examples, instead of assuming that the autonomous vehicle 100 will hop along the same selected trajectory at a future time, the training phase can be modified to account for the possibility that the hop may follow a different selected trajectory at a future time. Such modifications, taking into account the possibility of different future trajectories, can help the motion planner 330 find trajectories that are combinations of trajectory portions from a set of trajectories within the domain of the trajectory generator 332. Consequently, such modified training can cause the motion planner 330 to output more efficient vehicle maneuvers than would be possible with the set of trajectories available from the trajectory generator 332.

[0100] Training may be performed using the training RL processor 412 to learn the parameters of the following equation, which is slightly modified from equation (2) above:

[0101] V(s t ,τ t )=r t +γV(s t+1 ,TS(s t+1 )) (3)

[0102] Among them, TS(s t+1 ) is the environmental state s given the next time jump t+1 When t+1 is reached, the track selector 336 switches the output (ie, the selected track or the selected parameter set) at the next time t+1.

[0103] In this example, by avoiding the assumption that the trajectory is fixed for future jumps, the trajectory evaluator 334 can be trained to consider the possibility that the trajectory can change in future jumps. This expands the search space to include trajectories that are not strictly within the trajectory space of the trajectory generator 336 (e.g., it will include trajectories that change parameters at different times). As a result, the motion planner 330 can provide an output corresponding to a motion that is a combination of multiple trajectories (e.g., jumping along different trajectories at different times), which can be more complex and / or optimized than the single trajectory defined by the trajectory generator 336.

[0104] In some examples, the estimated evaluation values ​​assigned by the trajectory evaluator 334 may have statistical characteristics rather than scalar values. That is, given (s t ,τ t ) can be a probability distribution that reflects the characteristics of a random environment, since any given action in a random environment can only predict the probability of a reward. Distributional RL is a type of learning (s t ,τ t) method for the distribution of estimated evaluation values ​​compared to the mean in regular RL. Instead of assigning a scalar evaluation value to the candidate trajectories, the trajectory evaluator 334 can alternatively assign a probability distribution (e.g., defined by a statistical metric, such as mean, variance, maximum, minimum, etc.) to the candidate trajectories. Therefore, the trajectory selector 336 can be designed to make a selection based on one or more statistical metrics. For example, the selected trajectory can be the trajectory with the highest lower bound in the distribution of estimated evaluation values. This trajectory selection method may produce a more conservative (safer) motion plan in a non-deterministic environment with random events. In other words, the trajectory selector 336 selects the trajectory that produces the best worst-case outcome considering the probabilities of different events that may occur.

[0105] Figure 8 800 is a flow chart illustrating an exemplary method 800 for training the motion planner 330. Specifically, the method 800 may be used to train an agent of the trajectory estimator 334 (which may be implemented using a neural network such as a CNN or RNN). The method 800 may be implemented by a system (e.g., Figure 2 The training may be performed offline (e.g., in a simulator) or online (e.g., during actual driving).

[0106] At 802, a set of state data defining the current state (actual or simulated) of the autonomous vehicle and the environment is received. As described above, the state data may include data encoded as a 2D image. The state data may be, for example, an observed state received from the perception system 120.

[0107] At 804, a trajectory is selected for the current state (e.g., using trajectory selector 336). In some examples, the selected trajectory can be calculated using a set of parameters in a predefined trajectory generation function. A set of parameters is determined based on the current state (e.g., by applying a function to state data to generate the set of parameters), and the selected trajectory is then generated using the parameters determined in the trajectory generation function. In other examples, the selected trajectory can be selected from a set of trajectories generated by trajectory generator 332. Trajectory evaluator 334 calculates an evaluation value for each trajectory generated by trajectory generator 332, and trajectory selector 336 selects the trajectory with the highest calculated evaluation value as the selected trajectory. If a given generated trajectory is selected as the selected trajectory for the autonomous vehicle to follow, the evaluation value can represent a predicted reward. The evaluation value can be generated as a scalar value or as a set of statistical metrics that define a probability distribution of the probability evaluation values ​​(e.g., mean, variance, maximum, minimum, etc.). If the evaluation value is generated as a set of statistical metrics, trajectory selector 336 can select the selected trajectory based on one or more statistical selection criteria.

[0108] In some further examples, trajectory generator 332 may also generate a set of refined trajectories based on the selection, which may be evaluated by trajectory evaluator 334, and trajectory selector 336 may make an updated selection from the set of refined trajectories (such further refinement and selection may be conceptually equivalent to performing an iterative search).

[0109] At 806, for one time jump, the car 100 follows the selected trajectory at its current state (actual or simulated), and a reward is calculated based on the performance of the car 100. The reward is calculated according to a reward function designed for safe, comfortable, and fast (within the speed limit) driving.

[0110] At 808 , a set of state data is received that defines a next state (actual or simulated) of the autonomous vehicle and the environment at the next time hop (i.e., at t+1).

[0111] At 810, parameters (e.g., weights) of trajectory evaluator 334 (e.g., implemented as a CNN) are updated during training to better correspond to the discounted sum of expected future rewards. For example, this can be according to equation (2) or equation (3) above.

[0112] At 812, use the next state (ie, set s t =s t+1 ) updates the current state. Method 800 returns to step 804 to repeat the training with the updated state. Method 800 can continue until trajectory evaluator 334 is sufficiently trained (e.g., the weight values ​​of trajectory evaluator 334 converge).

[0113] The trained motion planner can then be implemented in the autonomous vehicle 100 for deployment in real-life scenarios. In some examples, training can be performed intermittently even after deployment.

[0114] In various examples described herein, systems and methods are provided that enable training a trajectory estimator to learn trajectory estimation for autonomous driving based on RL using a simple reward signal.

[0115] This disclosure describes examples for applying RL to train a trajectory evaluator agent to estimate an evaluation value based on one future time jump. Various methods of encoding input data (eg, input state) to implement a trajectory evaluator agent have been described.

[0116] Different methods of encoding tracks and defining the track for the next time jump have been described. For example, the track for the next time jump can be assumed to be the same as the track for the current time jump (just one time jump earlier), or it can be different (e.g., different tracks are selected based on different states of the next time jump).

[0117] While examples have been described in the context of autonomous vehicles, it should be understood that the present disclosure is not limited to applications in autonomous vehicles. For example, any trajectory planning application (e.g., for mobile robots, for semi-autonomous applications, for assisted driving applications, etc.) can benefit from the present disclosure. The present disclosure can be used to navigate structured or unstructured environments with static and / or dynamic obstacles.

[0118] Although the present disclosure describes methods and processes by steps performed in a certain order, one or more steps in the methods and processes may be omitted or changed as appropriate. Where appropriate, one or more steps may be performed in an order other than the order described.

[0119] Although the present disclosure is at least partially described in terms of methods, it will be understood by those skilled in the art that the present disclosure is also directed to various components for performing at least some aspects and features of the methods, whether through hardware components, software, or any combination thereof. Accordingly, the technical solutions of the present disclosure can be embodied in the form of software products. Suitable software products can be stored in pre-recorded storage devices or other similar non-volatile or non-transient computer-readable media, including DVDs, CD-ROMs, USB flash drives, removable hard drives, or other storage media. The software product includes instructions tangibly stored thereon that enable a processing device (e.g., a personal computer, a server, or a network device) to perform examples of the methods disclosed herein.

[0120] The present disclosure may be embodied in other specific forms without departing from the subject matter of the claims. The exemplary embodiments described are intended to be illustrative in all respects and not restrictive. Selected features from one or more of the above-described embodiments may be combined to create alternative embodiments not explicitly described, and features suitable for such combinations are understood to be within the scope of the present disclosure.

[0121] All values ​​and subranges within the disclosed ranges are also disclosed. Furthermore, although the systems, devices, and processes disclosed and illustrated herein may include a specific number of elements / components, these systems, devices, and assemblies may be modified to include more or fewer of such elements / components. For example, although any disclosed element / component may be referenced as a single quantity, the embodiments disclosed herein may be modified to include a plurality of such elements / components. The subject matter described herein is intended to cover and encompass all suitable technical variations.

Claims

1. A system for training a motion planner for an autonomous vehicle, characterized in that The system includes a processing unit configured to execute instructions such that the system: receiving, as input to a trajectory estimator agent of the motion planner, a first set of state data defining a current state of the autonomous vehicle and the environment being hopped at a current time; selecting a selected trajectory based on the current state; calculating a reward for the selected trajectory based on the performance of the selected trajectory under the current state; receiving a second set of state data, the second set of state data defining a next state of the autonomous vehicle and an environment to be transitioned to at a next time; updating parameters of the trajectory evaluator agent based on the current state, the selected trajectory, the calculated reward, and the next state, the next state comprising: a next time jump and an evaluation value of the next state or a next selected trajectory and an evaluation value of the next state; The parameters of the trajectory evaluator agent are updated to assign an evaluation value to the selected trajectory that reflects the calculated reward and expected performance of the selected trajectory in a future state.

2. The system according to claim 1, wherein: The first set of status data and the second set of status data each independently include status data encoded in the form of a 2D image.

3. The system according to claim 1 or 2, characterized in that The parameters of the trajectory estimator agent are updated according to the following equations: V(s t ,τ t )←r t +γV(s t+1 ,τ t+1 ) Among them, s t is the current state at the current time jump t, τ t is the selected trajectory, r t is the computational reward, V(s t ,τ t ) is the evaluation value of the selected trajectory at the current time jump and the current state, t+1 is the next time jump, V(s t+1 ,τ t+1 ) is the evaluation value of the selected trajectory at the next time jump and the next state, and γ is the discount factor.

4. The system according to claim 1 or 2, characterized in that The parameters of the trajectory estimator are updated according to the following equations: V(s t ,τ t )←r t +γV(s t+1 ,TS(s t+1 )) Among them, s t is the current state at the current time jump t, τ t is the selected trajectory, r t is the computational reward, V(s t ,τ t ) is the evaluation value of the selected trajectory at the current time jump and the current state, t+1 is the next time jump, TS(s t+1 ) is the next selected trajectory to jump at the next time, V(s t+1 ,TS(s t+1 )) is the evaluation value of the next selected trajectory and the next state, and γ is the discount factor.

5. The system according to claim 1 or 2, characterized in that The estimates are generated as a set of statistical measures defining a probability distribution of the probability estimates.

6. The system according to claim 5, characterized in that The selected trajectory is selected according to a selection criterion based on one or more statistical metrics.

7. The system according to claim 1 or 2, characterized in that The set of data defining the candidate trajectories is a set of parameters defining the candidate trajectories according to the trajectory generation function.

8. The system according to claim 7, characterized in that The processing unit is configured to execute instructions to further enable the system to: The selected trajectory is generated from the set of parameters according to the trajectory generation function.

9. The system according to claim 1 or 2, characterized in that The selected trajectory is defined by a set of 2D images defining waypoints of a candidate trajectory over a plurality of time hops.

10. A method for training a motion planner for an autonomous vehicle, characterized in that The method comprises: receiving, as input to a trajectory estimator of the motion planner, a first set of state data defining a current state of the autonomous vehicle and the environment being transitioned at a current time; selecting a selected trajectory based on the current state; calculating a reward for the selected trajectory based on the performance of the selected trajectory under the current state; receiving a second set of state data, the second set of state data defining a next state of the autonomous vehicle and an environment to be transitioned to at a next time; Parameters of the trajectory evaluator agent are updated based on the current state, the selected trajectory, the calculated reward, and a next state, the next state comprising: a next time jump and an evaluation value of the next state or a next selected trajectory and an evaluation value of the next state; the parameters of the trajectory evaluator agent are updated to assign the selected trajectory an evaluation value that reflects the calculated reward and expected performance of the selected trajectory in a future state.

11. The method according to claim 10, characterized in that The first set of status data and the second set of status data each independently include status data encoded in the form of a 2D image.

12. The method according to claim 10 or 11, characterized in that The parameters of the trajectory estimator are updated according to the following equations: V(s t ,τ t )←r t +γV(s t+1 ,τ t+1 ) Among them, s t is the current state at the current time jump t, τ t is the selected trajectory, r t is the computational reward, V(s t ,τ t ) is the evaluation value of the selected trajectory at the current time jump and the current state, t+1 is the next time jump, V(s t+1 ,τ t+1 ) is the evaluation value of the selected trajectory at the next time jump and the next state, and γ is the discount factor.

13. The method according to claim 10 or 11, characterized in that The parameters of the trajectory estimator are updated according to the following equations: V(s t ,τ t )←r t +γV(s t+1 ,TS(s t+1 )) Among them, s t is the current state at the current time jump t, τ t is the selected trajectory, r t is the computational reward, V(s t ,τ t ) is the evaluation value of the selected trajectory at the current time jump and the current state, t+1 is the next time jump, TS(s t+1 ) is the next selected trajectory to jump at the next time, V(s t+1 ,TS(s t+1 )) is the evaluation value of the next selected trajectory and the next state, and γ is the discount factor.

14. The method according to claim 10 or 11, characterized in that The estimates are generated as a set of statistical measures defining a probability distribution of the probability estimates.

15. The method according to claim 14, characterized in that The selected trajectory is selected according to a selection criterion based on one or more statistical metrics.

16. The method according to claim 10 or 11, characterized in that The set of data defining the candidate trajectories is a set of parameters defining the candidate trajectories according to the trajectory generation function.

17. The method according to claim 16, characterized in that Also includes: The selected trajectory is generated from the set of parameters according to the trajectory generation function.

18. The method according to claim 10 or 11, characterized in that The selected trajectory is defined by a set of 2D images defining waypoints of a candidate trajectory over a plurality of time hops.

19. A computer-readable medium, characterized in that Storing instructions for execution by a processing unit of a system for training a motion planner for an autonomous vehicle, the instructions, when executed, causing the system to perform the method according to any one of claims 11 to 18.

20. A computer program, characterized in that comprising instructions which, when executed by a processing unit of a system for training a motion planner for an autonomous vehicle, cause the system to perform the method according to any one of claims 11 to 18.

Images