Autonomous vehicle operation method, electronic device, and computer readable medium

Abstract

The present disclosure provides an automatic driving vehicle operation method, and relates to the field of automatic driving. The automatic driving vehicle operation method according to the embodiment of the present disclosure comprises the following operation: in one embodiment, when the automatic driving vehicle enters a driving scene, the automatic driving vehicle can call a map-based scene inspector to determine the type of the scene, and call a corresponding neural network model to generate a parameter set of the scene based on real-time environmental conditions and vehicle state information. The scene parameter set can be a set of additional constraints for configuring the automatic driving vehicle to drive in a driving mode corresponding to the scene.

Description

Technical Field

[0001] The embodiments of this disclosure generally relate to operating autonomous vehicles. More specifically, the embodiments of this disclosure relate to dynamically generating scene parameters based on real-time environmental data and real-time vehicle state information. Background Technology

[0002] Autonomous driving vehicles (ADVs) can reduce some driving-related tasks for passengers, especially the driver, when operating in automatic mode. When the vehicle is running in automatic mode, onboard sensors can be used to navigate the vehicle to different locations, thus minimizing human-machine interaction or allowing the vehicle to operate without passengers.

[0003] When autonomous vehicles travel on roads, they encounter various driving scenarios, such as left turns, right turns, intersections, and straight lanes. Each driving scenario (also called a scene) requires the autonomous vehicle to drive differently based on the characteristics of the driving situation. For example, when driving at an intersection, the autonomous vehicle needs to consider more information than when driving in a straight lane because intersections have more complex road and traffic conditions. Therefore, autonomous vehicles can use different parameters (e.g., when to decelerate, when to accelerate, etc.) as additional constraints to generate trajectories to navigate through different scenarios.

[0004] However, using the same fixed set of parameters for the same type of scenario, without considering the scenario location or the time when the autonomous vehicle enters the scenario, leads to relatively low efficiency. For example, the environmental conditions at an intersection on one road may differ from those at an intersection on another road. Furthermore, the environmental conditions for the same scenario on the same road may differ at different times of the day. For example, traffic is busier during peak hours than during off-peak hours. Summary of the Invention

[0005] This disclosure provides a method for operating an autonomous vehicle, a computer-readable medium, and an electronic device.

[0006] According to one aspect of this disclosure, a computer-implemented method for operating an autonomous vehicle (ADV) is provided, comprising:

[0007] The scene inspector on the autonomous vehicle determines the scene that the autonomous vehicle has entered in the road segment;

[0008] By using autonomous vehicles, a trained neural network model corresponding to the scene is invoked. This trained neural network model generates a parameter set for the scene based on environmental data and the vehicle's state information.

[0009] Operate an autonomous vehicle to drive through a scene in a driving mode defined by a set of scene parameters.

[0010] According to another aspect of the embodiments of this disclosure, a non-transitory computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform the following operations for operating an autonomous driving vehicle (ADV):

[0011] Identify the scenarios in which autonomous vehicles have entered the road segment;

[0012] The trained neural network model corresponding to the scene is invoked, wherein the trained neural network model generates a parameter set for the scene based on environmental data of the scene and vehicle state information of the autonomous vehicle; and

[0013] Operate an autonomous vehicle to drive through a scene in a driving mode defined by a set of scene parameters.

[0014] According to another aspect of the embodiments of this disclosure, an electronic device is provided, comprising:

[0015] Processor; and

[0016] The memory coupled to the processor stores instructions that, when executed by the processor, cause the processor to perform the following actions to operate the autonomous vehicle (ADV):

[0017] Identify the scenarios in which autonomous vehicles have entered the road segment;

[0018] The trained neural network model corresponding to the scene is invoked, wherein the trained neural network model generates a parameter set for the scene based on environmental data of the scene and vehicle state information of the autonomous vehicle; and

[0019] Operate an autonomous vehicle to drive through a scene in a driving mode defined by a set of scene parameters.

[0020] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0021] Embodiments of this disclosure are shown in the accompanying drawings by way of example rather than limitation, wherein similar reference numerals denote similar elements.

[0022] Figure 1 This is a block diagram illustrating a networked system according to one embodiment.

[0023] Figure 2This is a block diagram illustrating an example of an autonomous vehicle according to one embodiment.

[0024] Figure 3A and Figure 3B This is a block diagram illustrating an example of an autonomous driving system used with an autonomous vehicle according to one embodiment.

[0025] Figure 4 This is a block diagram illustrating an example of a decision-making and planning system according to one embodiment.

[0026] Figure 5 A planning module for dynamically generating scene parameters is shown according to one embodiment.

[0027] Figure 6 This illustrates how fixed scene parameters are generated according to one embodiment.

[0028] Figure 7 A planning module according to another embodiment is shown.

[0029] Figure 8 This is a flowchart illustrating the operation process of an autonomous vehicle according to one embodiment. Specific Implementation

[0030] Various embodiments and aspects of this disclosure will be described with reference to the following details, and the accompanying drawings will illustrate various embodiments. The following description and drawings are illustrative and should not be construed as limiting the scope of this disclosure. Numerous specific details have been described to provide a thorough understanding of the various embodiments of this disclosure. However, in some cases, well-known or conventional details have not been described in order to concisely illustrate the embodiments of this disclosure.

[0031] The phrase "an embodiment" or "embodiment" used in the specification means that a feature, structure, or characteristic described in connection with that embodiment may be included in at least one embodiment of this disclosure. The phrase "in an embodiment" appearing in different locations in the specification does not necessarily refer to the same embodiment.

[0032] According to some embodiments, systems, methods, and media for dynamically generating scene parameters for automated vehicle (ADV) scenarios are described. In one embodiment, when an ADV enters a driving scenario, the ADV may invoke a map-based scene inspector to determine the type of scenario and invoke a corresponding neural network model to generate a set of parameters for the scenario based on real-time environmental conditions (e.g., traffic volume) and vehicle state information (e.g., speed). The scene parameter set may be a set of additional constraints for configuring the ADV to drive in a driving mode corresponding to the scenario.

[0033] In one embodiment, the scene inspector can determine the type of scene the autonomous vehicle has entered based on the vehicle's current location, map information around the current location, information about the route the vehicle is taking, and one or more fixed parameters from a fixed parameter set of the scene. Specifically, the fixed parameter set of the scene is generated by a trained neural network model based on historical environmental data and historical vehicle state information.

[0034] In one embodiment, the environmental data of the scene includes the traffic volume of the scene, and the vehicle status information of the autonomous vehicle includes the speed of the autonomous vehicle.

[0035] In one embodiment, the scenario parameter set represents a set of additional constraints used by the autonomous vehicle when generating its planned trajectory, reflecting human-like driving behavior as it traverses the scenario. Therefore, different scenario parameter sets can exist depending on the specific scenario environment and the time the autonomous vehicle enters the scenario.

[0036] In one embodiment, a scenario is one of multiple scenarios predefined based on map information. These scenarios include intersections, yield signs, stop signs, single-lane right turns, single-lane left turns, multi-lane right turns, and multi-lane left turns.

[0037] The above embodiments do not exhaustively cover all aspects of the invention. It can be understood that the invention includes all embodiments that can be derived from all suitable combinations of the embodiments outlined above and the embodiments disclosed below.

[0038] autonomous vehicles

[0039] Figure 1 This is a block diagram illustrating an autonomous driving network configuration according to an embodiment of the present disclosure. See also Figure 1 Network configuration 100 includes an autonomous vehicle (autonomous vehicle) 101, which can communicate with one or more servers 103-104 via network 102. While only one autonomous vehicle is shown in the figure, multiple autonomous vehicles can be coupled to each other and / or to servers 103-104 via network 102. Network 102 can be any type of wired or wireless network, such as a local area network (LAN), a wide area network (WAN) such as the Internet, a cellular network, a satellite network, or a combination thereof. Servers 103-104 can be any type of server or server cluster, such as a web or cloud server, an application server, a backend server, or a combination thereof. Servers 103-104 can be data analytics servers, content servers, traffic information servers, map and point of interest (MPOI) servers, or location servers, etc.

[0040] An autonomous vehicle is a vehicle configurable to operate in an automatic mode, in which it can navigate with minimal or no driver input. Such an autonomous vehicle may include a sensor system with one or more sensors for detecting environmental information while the vehicle is operating. The vehicle and its associated controllers use the detected information to navigate in this environment. The autonomous vehicle 101 can operate in manual, fully automatic, or partially automatic modes.

[0041] In one embodiment, the autonomous vehicle 101 includes, but is not limited to, an Autonomous Driving System (ADS) 110, a vehicle control system 111, a wireless communication system 112, a user interface system 113, and a sensor system 115. The autonomous vehicle 101 may also include some common components found in ordinary vehicles, such as an engine, wheels, steering wheel, and transmission. These components can be controlled via the vehicle control system 111 and / or the autonomous driving system 110 using various communication signals and / or commands, such as acceleration signals or commands, deceleration signals or commands, steering signals or commands, and braking signals or commands.

[0042] Components 110-115 can be communicatively coupled to each other via interconnects, buses, networks, or combinations thereof. For example, components 110-115 can be communicatively coupled to each other via a Controller Area Network (CAN) bus. The CAN bus is a vehicle bus standard designed to enable communication between microcontrollers and devices in masterless applications. The CAN bus is a message-based protocol originally designed for multiplexing electrical wiring in automobiles, but it is also used in many other environments.

[0043] See Figure 2In one embodiment, the sensor system 115 includes, but is not limited to, one or more cameras 211, a Global Positioning System (GPS) unit 212, an Inertial Measurement Unit (IMU) 213, a radar unit 214, and a Light Detection and Range (LIDAR) unit 215. The GPS system 212 may include a transceiver for providing information related to the location of the autonomous vehicle. The IMU unit 213 may sense changes in the position and orientation of the autonomous vehicle based on inertial acceleration. The radar unit 214 may represent a system for sensing objects within the local environment of the autonomous vehicle using radio signals. In some embodiments, in addition to sensing objects, the radar unit 214 may also sense the velocity and / or heading of objects. The LiDAR unit 215 may use lasers to sense objects in the environment in which the autonomous vehicle is located. The LiDAR unit 215 may include one or more laser sources, a laser scanner, and one or more detectors, as well as other system components. The camera 211 may include one or more devices for capturing images of the environment surrounding the autonomous vehicle. The camera 211 may be a camera and / or a video camera. The camera may be mechanically movable, for example, by mounting it on a rotating and / or tilting platform.

[0044] Sensor system 115 may also include other sensors, such as sonar sensors, infrared sensors, steering sensors, throttle sensors, brake sensors, and audio sensors (e.g., microphones). Audio sensors can be used to capture sound from the environment surrounding the autonomous vehicle. Steering sensors can be used to sense the steering angle of the steering wheel, wheels, or a combination thereof. Throttle and brake sensors sense the throttle and brake positions of the vehicle, respectively. In some cases, the throttle and brake sensors can be integrated into an integrated throttle / brake sensor.

[0045] In one embodiment, the vehicle control system 111 includes, but is not limited to, a steering unit 201, a throttle unit 202 (also referred to as an acceleration unit), and a braking unit 203. The steering unit 201 is used to adjust the direction or heading of the vehicle. The throttle unit 202 is used to control the speed of the engine or throttle, thereby controlling the speed and acceleration of the vehicle. The braking unit 203 decelerates the vehicle by providing friction to the wheels or tires. It should be noted that... Figure 2 The components shown can be implemented as hardware, software, or a combination thereof.

[0046] See you again Figure 1The wireless communication system 112 allows the autonomous vehicle 101 to communicate with external systems, such as devices, sensors, and other vehicles. For example, the wireless communication system 112 can communicate directly with one or more devices, or via a communication network such as servers 103-104 on network 102. The wireless communication system 112 can utilize any cellular communication network or wireless local area network (WLAN), such as WiFi, to communicate with another component or system. The wireless communication system 112 can communicate directly with devices (e.g., mobile devices of passengers within vehicle 101, display devices, speakers), for example, using infrared links, Bluetooth, etc. The user interface system 113 may be some peripheral devices implemented within vehicle 101, including, for example, a keyboard, touchscreen display, microphone, and speaker.

[0047] Some or all of the functions of the autonomous vehicle 101 can be controlled or managed by the autonomous driving system 110, especially when operating in autonomous driving mode. The autonomous driving system 110 includes the necessary hardware (e.g., processor, memory, storage) and software (e.g., operating system, planning and route selection program) to receive information from the sensor system 115, control system 111, wireless communication system 112, and / or user interface system 113, process the received information, plan a route or path from the origin to the destination, and then drive the vehicle 101 based on the planning and control information. Alternatively, the autonomous driving system 110 can be integrated with the vehicle control system 111.

[0048] For example, a passenger can specify the start and destination of their trip via a user interface. The autonomous driving system 110 obtains trip-related data. For instance, the autonomous driving system 110 can obtain location and route data from a map and point-of-interest (POI) server, which may be part of servers 103-104. The location server provides location services, and the map and POI server provides map services and points of interest for some locations. Alternatively, this location, map, and POI information can be cached locally in the persistent storage of the autonomous driving system 110.

[0049] As the autonomous vehicle 101 moves along the route, the autonomous driving system 110 can also obtain real-time traffic information from a Traffic Information System or Server (TIS). It should be noted that servers 103-104 can be operated by a third-party entity. Alternatively, the functions of servers 103-104 can be integrated into the autonomous driving system 110. Based on real-time traffic information, map and point-of-interest information, location information, and real-time local environmental data (such as obstacles, objects, and nearby vehicles) detected or sensed by the sensor system 115, the autonomous driving system 110 can, for example, plan the optimal route according to the planned route via the control system 111 and drive the vehicle 101 to safely and efficiently reach the designated destination.

[0050] Server 103 may be a data analysis system for providing data analysis services to various clients. In one embodiment, data analysis system 103 includes a data collector 121 and a machine learning engine 122. Data collector 121 collects driving statistics 123 from various vehicles, such as autonomous vehicles or conventional vehicles driven by human drivers. Driving statistics 123 include information representing issued driving commands (e.g., accelerator, brake, steering commands) and vehicle responses (e.g., speed, acceleration, deceleration, direction) captured by vehicle sensors at different points in time. Driving statistics 123 may also include information describing the driving environment at different points in time, such as routes (including origin and destination locations), maps and points of interest, road conditions, weather conditions, etc.

[0051] The machine learning engine 122 can generate or train a set of rules, algorithms, and / or prediction models 124 for various purposes based on driving statistics 123. The algorithms 124 can then be uploaded to the autonomous vehicle for real-time use during autonomous driving.

[0052] Figure 3A and Figure 3B This is a block diagram illustrating an example of an autonomous driving system used with an autonomous vehicle according to one embodiment. System 300 can be implemented as follows: Figure 1 This is part of the autonomous vehicle 101, including but not limited to the autonomous driving system 110, the control system 111, and the sensor system 115. See also Figure 3A and Figure 3B The autonomous driving system 110 includes, but is not limited to, a positioning module 301, a perception module 302, a prediction module 303, a decision-making module 304, a planning module 305, a control module 306, and a route module 307.

[0053] Some or all of modules 301-307 can be implemented as software, hardware, or a combination thereof. For example, these modules can be installed in persistent storage device 352, loaded into memory 351, and executed by one or more processors (not shown). It should be noted that some or all of these modules can be integrated with… Figure 2 Some or all of the modules of the vehicle control system 111 are communicatively coupled or integrated together. Some of the modules in modules 301 to 307 can be integrated into an integrated module.

[0054] The positioning module 301 determines the current location of the autonomous vehicle 300 (e.g., using GPS unit 212) and manages any data related to the user's trip or route. The positioning module 301 (also called the map and route module) manages any data related to the user's trip or route. The user can log in, for example, through a user interface and specify the start and destination of the trip. The positioning module 301 communicates with other components of the autonomous vehicle 300, such as map and route data 311, to obtain trip-related data. For example, the positioning module 301 can obtain location and route data from a location server and a map and point-of-interest (POI) server. The location server provides location services, and the map and POI server provides map services and points of interest for some locations, which can be cached as part of the map and route data 311. As the autonomous vehicle 300 moves along the route, the positioning module 301 can also obtain real-time traffic information from a traffic information system or server.

[0055] The perception module 302 can determine the perceived information of the surrounding environment based on the sensor data provided by the sensor system 115 and the positioning information obtained by the positioning module 301. The perceived information can represent information about the vehicle's surroundings that a normal driver can perceive while driving. The perceived information may include, for example, lane configurations in the form of objects, traffic light signals, the relative positions of other vehicles, pedestrians, buildings, crosswalks, or other traffic-related signs (e.g., stop signs, yield signs). Lane configurations include information describing one or more lanes, such as lane shape (e.g., straight or curved), lane width, number of lanes on the road, one-way or two-way lanes, merging or dividing lanes, lane departure, etc.

[0056] The perception module 302 may include a computer vision system or the functionality of a computer vision system to process and analyze images captured by one or more cameras to identify objects and / or features in the environment of the autonomous vehicle. Objects may include traffic signals, road boundaries, other vehicles, pedestrians and / or obstacles, etc. The computer vision system may utilize object recognition algorithms, video tracking, and other computer vision techniques. In some embodiments, the computer vision system may map the environment, track objects, and estimate the velocity of objects, etc. The perception module 302 may also detect objects based on additional sensor data provided by other sensors such as radar and / or lidar.

[0057] For each object, prediction module 303 predicts the object's behavior in the environment. This prediction is performed based on perception data, which senses the driving environment at a given point in time based on a set of map / route information 311 and traffic rules 312. For example, if the object is a vehicle traveling in the opposite direction and the current driving environment includes an intersection, prediction module 303 will predict that the vehicle is likely to go straight or turn. If the perception data indicates that there are no traffic lights at the intersection, prediction module 303 can predict that the vehicle must come to a complete stop before entering the intersection. If the perception data indicates that the vehicle is currently in a dedicated left-turn lane or a dedicated right-turn lane, prediction module 303 can predict that the vehicle is more likely to turn left or right, respectively.

[0058] For each object, decision module 304 makes a decision related to the object. For example, given an object (e.g., another vehicle on an intersecting road) and its metadata (e.g., speed, direction, turning angle), decision module 304 decides how to deal with the object (e.g., overtake, yield, stop, pass). Decision module 304 can make these decisions based on a set of rules, such as traffic rules or driving rules 312, which can be stored in persistent storage device 352.

[0059] Route module 307 provides one or more routes or paths from a starting point to a destination. For a given trip from a starting location to a destination location, received from a user, for example, route module 307 obtains route and map information 311 and determines all possible routes or paths from the starting location to the destination location. Route module 307 can generate a reference line in the form of a topographic map for each route it determines from the starting location to the destination location. The reference line refers to the ideal route or path in the absence of interference such as other vehicles, obstacles, or traffic conditions. That is, if there are no other vehicles, pedestrians, or obstacles on the road, the autonomous vehicle should follow the reference line exactly or approximately. The topographic map is then provided to decision module 304 and / or planning module 305. Decision module 304 and / or planning module 305 examine all possible routes to select and modify one of the optimal routes based on other data provided by other modules, such as traffic conditions from positioning module 301, driving environment perceived by perception module 302, and traffic conditions predicted by prediction module 303. The actual path or route used to control an autonomous vehicle may be close to or different from the reference line provided by the route module 307, depending on the specific driving environment at that point in time.

[0060] The planning module 305 can plan the path or route of the autonomous vehicle and driving parameters (such as distance, speed, and / or turning angle) based on the decisions made for each perceived object and using reference lines provided by the route module 307. That is, for a given object, the decision module 304 decides what to do for that object, while the planning module 305 decides how to do it. For example, for a given object, the decision module 304 may decide to pass through the object, while the planning module 305 may determine whether to pass from the left or right side of the object. The planning module 305 generates planning and control data, including information describing how the vehicle 101 will move in the next movement cycle (e.g., the next route / path segment). For example, the planning and control data may instruct the vehicle 101 to move 10 meters at a speed of 30 miles per hour (mph) and then change to the right lane at a speed of 25 miles per hour.

[0061] Control module 306 controls and drives the autonomous vehicle based on planning and control data, according to a route or path defined by planning and control information, by sending appropriate commands or signals to vehicle control system 111. The planning and control data includes sufficient information to drive the vehicle from one point to another along the route or path at different times using appropriate vehicle settings or driving parameters (e.g., throttle, braking, steering commands).

[0062] In one embodiment, the planning phase is executed over multiple planning cycles, also known as driving cycles, for example, time intervals of 100 milliseconds (ms). For each planning cycle or driving cycle, one or more control commands can be issued based on the planning and control data. That is, for every 100 milliseconds, the planning module 305 plans, for example, the next route segment or path segment including the target location and the time required for the autonomous vehicle to reach the target location. Alternatively, the planning module 305 may also specify specific speeds, directions, and / or steering angles, etc. In one embodiment, the planning module 305 plans, for example, the next predetermined time interval of 5 seconds. For each planning cycle, the planning module 305 plans the target location for the current cycle (e.g., the next 5 seconds) based on the target location planned in the previous cycle. Subsequently, the control module 306 generates one or more control commands (e.g., throttle, brake, steering control commands) based on the planning and control data of the current cycle.

[0063] It is important to note that the decision-making module 304 and the planning module 305 can be integrated into a single module. The decision-making module 304 / planning module 305 may include a navigation system or the functionality of a navigation system to determine the driving path of the autonomous vehicle. For example, the navigation system may determine a series of speeds and directional headings, causing the autonomous vehicle to move along a path that substantially avoids perceived obstacles, while also typically guiding the autonomous vehicle along a lane-based path leading to the final destination. The destination can be set via user interface system 113 based on user input. The navigation system can dynamically update the driving path while the autonomous vehicle is in operation. The navigation system can combine data from a GPS system and one or more maps to determine the driving path of the autonomous vehicle.

[0064] Figure 4 This is a block diagram illustrating an example of a decision-making and planning system according to one embodiment. System 400 can be implemented as follows: Figure 3A and Figure 3B It is part of the autonomous driving system 300, used to perform path planning and speed planning operations. See also Figure 4 The decision and planning system 400 (also known as the planning and control system or module, or PnC system or module) includes a route module 307, a positioning / sensing data 401, a path decision module 403, a speed decision module 405, a path planning module 407, a speed planning module 409, an aggregator 411, and a trajectory calculator 413, etc.

[0065] The route decision module 403 and the speed decision module 405 can be implemented as part of the decision module 304. In one embodiment, the route decision module 403 may include a route state machine, one or more route traffic rules, and a station-lateral map generator. The route decision module 403 may use dynamic programming to generate rough route curves as initial constraints for the route / speed planning modules 407 and 409.

[0066] In one embodiment, the path state machine includes at least three states: cruising, lane-changing, and / or idle. The path state machine provides previous planning results and important information, such as whether the autonomous vehicle is cruising or changing lanes. The path traffic rules can be... Figure 3A The driving / traffic rules 312 are part of the driving / traffic rules, including traffic rules that may affect the results of the path decision module. For example, the path traffic rules may include traffic information such as nearby construction signs, which the autonomous vehicle can avoid lanes with construction signs. The path decision module 403 can determine how to deal with perceived obstacles (i.e., ignore, overtake, yield, stop, or pass) based on the state, traffic rules, reference lines provided by the route module 307, and obstacles perceived by the autonomous vehicle's perception module 302, as part of the coarse path curve.

[0067] For example, in one embodiment, a coarse path curve is generated based on the path curvature and the distance between a reference line and / or a reference point and an obstacle, using a cost function comprised of costs. Points on the reference line are selected and moved to the left or right of the reference line as candidate moves representing path candidates. Each candidate move has an associated cost. Using optimal cost dynamic programming, the associated costs of candidate moves for one or more points on the reference line can be solved sequentially, point by point.

[0068] In one embodiment, a Station-Lateral (SL) graph generator (not shown) generates an SL graph as part of a coarse path curve. The SL graph is a two-dimensional geometric graph (similar to an xy-coordinate plane) that includes obstacle information perceived by the autonomous vehicle. The path decision module 403 can arrange an autonomous vehicle path that conforms to obstacle decisions based on the SL graph. Dynamic programming (also known as dynamic optimization) is a mathematical optimization method that decomposes the problem to be solved into a series of value functions, each of which is solved only once, and the solution is stored. When the same value function appears again, the solution is not recalculated; instead, the previously calculated solution is retrieved, thus saving computation time.

[0069] The speed decision module 405 or speed decision process includes a speed state machine, speed traffic rules, and a station-time map generator (not shown). The speed decision module 405 or speed decision process can utilize dynamic programming to generate a coarse speed curve as initial constraints for the path / speed planning modules 407 and 409. In one embodiment, the speed state machine includes at least two states: an acceleration state and / or a deceleration state. The speed traffic rules can be... Figure 3A The driving / traffic rules 312 are part of the speed decision module and include traffic rules that can affect the results of the speed decision module. For example, speed traffic rules may include traffic information such as red / green traffic lights, another vehicle at an intersection, etc. The speed decision module 405 can generate a coarse speed curve based on the state of the speed state machine, the speed traffic rules, the coarse path curve / SL diagram generated by the decision module 403, and perceived obstacles to control the timing of acceleration and / or deceleration of the autonomous vehicle. The SL diagram generator can generate a station-time (ST) diagram as part of the coarse speed curve.

[0070] In one embodiment, the path planning module 407 includes one or more SL graphs, a geometric smoother, and a path cost module (not shown). The SL graph may include a station-lateral graph generated by the SL graph generator of the path decision module 403. The path planning module 407 may use a coarse path curve (e.g., a station-lateral graph) as an initial constraint to recalculate the optimal reference line using quadratic programming. Quadratic programming (QP) involves minimizing or maximizing an objective function (e.g., a quadratic function with multiple variables) constrained by boundaries, linear equality, and inequalities.

[0071] One difference between dynamic programming and quadratic programming is that quadratic programming optimizes all candidate moves for all points on the reference line together. A geometric smoother can apply smoothing algorithms (such as B-splines or regression) to the output station-lateral plot. The path cost module can recalculate the reference line using a path cost function, for example, by using QP optimization performed by a QP module (not shown), to optimize the total cost of candidate moves for the reference points. For example, in one embodiment, the total path cost function can be defined as follows: Path cost = ∑ points (heading) 2 +∑ p oints(curvature) 2 +∑ p oints(distance) 2 ,

[0072] Here, path cost is the sum of all points on the reference line, heading represents the difference in radial angle (e.g., direction) between a point and the reference line, curvature represents the difference in curvature between the curve formed by these points and the reference line, and distance represents the lateral (perpendicular to the direction of the reference line) distance between a point and the reference line. In some embodiments, distance represents the distance between a point and the target position or intermediate point of the reference line. In another embodiment, curvature is the change in curvature value of the curve formed at adjacent points. It should be noted that points equidistant from adjacent points can be selected as points on the reference line. The path cost module can recalculate the reference line based on the path cost, for example, by using a quadratic programming module to optimize and minimize the path cost.

[0073] The velocity planning module 409 includes a station-time graph, a sequence smoother, and a velocity cost module. The station-time graph may include an ST graph generated by the ST graph generator of the velocity decision module 405. The velocity planning module 409 can compute an optimal station-time curve using a coarse velocity curve (e.g., the station-time graph) and results from the path planning module 407 as initial constraints. The sequence smoother can apply a smoothing algorithm (e.g., B-spline or regression) to the time series of points. The velocity cost module can recompile the ST graph using a velocity cost function to optimize the total cost of movement candidates (e.g., acceleration / deceleration) at different time points.

[0074] For example, in one embodiment, the total speed cost function is as follows:

[0075] Speed ​​cost = ∑ points (speed′) 2 +∑ po ints(speed") 2 +(distance) 2 ,

[0076] Here, speed cost is the total of all time regression points, speed′ represents the acceleration value or the cost of changing speed between two adjacent points, speed″ represents the jump amount, or the derivative of the acceleration value or the cost of changing acceleration between two adjacent points, and distance represents the distance between the ST point and the destination location. The speed cost module, for example, uses a QP module to optimize and minimize speed cost to calculate the station-time graph.

[0077] Aggregator 411 performs the function of aggregating path and velocity planning results. For example, in one embodiment, aggregator 411 can combine a two-dimensional ST map and SL map into a three-dimensional SLT map. In another embodiment, aggregator 411 can interpolate (or fill in additional points) based on two consecutive points on the SL reference line or ST curve. In yet another embodiment, aggregator 411 can convert reference points from (S, L) coordinates to (x, y) coordinates. Trajectory generator 413 can calculate the final trajectory to control the autonomous vehicle 101. For example, trajectory generator 413 can calculate a list of (x, y, T) points based on the SLT map provided by aggregator 411, indicating the time at which the autonomous vehicle should pass through specific (x, y) coordinates.

[0078] Therefore, the path decision module 403 and the speed decision module 405 are used to generate rough path curves and rough speed curves, taking into account obstacles and / or traffic conditions. Given all path and speed decisions regarding obstacles, the path planning module 407 and the speed planning module 409 utilize QP planning to optimize the rough path curves and rough speed curves based on obstacles, thereby generating an optimal trajectory with minimum path cost and / or speed cost.

[0079] Dynamic scene parameters

[0080] Figure 5 A planning module 305 for generating scene parameters according to an embodiment is shown.

[0081] like Figure 5 As shown, the planning module 305 may include multiple neural network models 505, each corresponding to one of multiple predefined scenarios. Each scenario is a driving environment determined based on map information. For example, predefined scenarios may include intersections, yield signs, stop signs, single-lane right turns, single-lane left turns, multi-lane right turns, and multi-lane left turns.

[0082] Each scenario can be associated with a different set of parameters, which can be used to specify the boundaries of the scenario and the expected behavior of the autonomous vehicle after entering the scenario.

[0083] For example, in an intersection scenario, the boundary parameter could be a distance of 12 meters from the traffic lights in the scenario. This means that when the distance between the autonomous vehicle and the traffic lights is less than 12 meters, the autonomous vehicle is considered to have entered the intersection scenario. Still using the intersection scenario as an example, parameters specifying the behavior of the autonomous vehicle in the scenario could include the vehicle's speed, acceleration, the time point when deceleration will occur, the time point when acceleration will occur, and the steering wheel angle.

[0084] The parameter set for each predetermined scenario can be fixed, regardless of the scenario location and the time when the autonomous vehicle enters the scenario. For example, if the boundary of an intersection is defined as 12 meters or closer to a traffic light, then regardless of which road the autonomous vehicle is on or when (e.g., during peak or off-peak hours) it encounters the intersection scenario, the planning module 305 of the autonomous vehicle 101 will consider the autonomous vehicle to have entered the scenario when it is 12 meters or closer to the traffic light at the intersection, and will operate according to the fixed parameters of the scenario.

[0085] like Figure 5 As shown, fixed scene parameters 515 represent multiple fixed scene parameter sets for multiple predefined scenarios of the autonomous vehicle 101. The fixed scene parameter set for each predefined scenario can be generated by one of multiple neural network models 505, wherein the neural network model 505 is trained based on training data collected from manually driven vehicles driving through the scenario.

[0086] In one embodiment, if the default set of scene parameters is not overridden or updated, the fixed set of scene parameters for a certain type of scene can be the default parameters when an autonomous vehicle enters that scene.

[0087] A fixed set of parameters for a scene can be generated using a neural network model corresponding to that scene. The neural network model can be trained using historical vehicle state information and environmental conditions extracted from log files generated by multiple manually driven vehicles traversing multiple scenes at different times and locations. In one embodiment, this can be achieved from... Figure 1 Training data is obtained from driving statistics collected by the data collector 121 shown. From this, a corresponding neural network model can generate a parameter set that simulates the behavior of an ordinary person passing through this scenario.

[0088] like Figure 5 As shown, the planning module 315 may include a map-based scene inspector 501. The scene inspector 501 can determine the type of scene the autonomous vehicle has entered based on map information 509, route information 511, vehicle position 513, and one or more fixed scene parameters (e.g., boundary information) from each fixed scene parameter set of fixed scene parameters 515.

[0089] In one embodiment, the map-based scene inspector 501 may be a convolutional neural network (CNN) model or other type of neural network model, which is trained to classify different types of scenes and determine whether the autonomous vehicle 101 is located in a certain scene based on the vehicle position 513 and scene boundary parameters.

[0090] Once the planning module 315 determines that it has entered a certain scenario, it can select a trained neural network model for that scenario from all existing neural network models 505. The selected neural network model can then be used to generate an updated scenario parameter set 507 based on the current environmental data 502 and the current vehicle state information 504 of the autonomous vehicle 101. The speed planner 517 and path planner 519 in the planning module can then use the updated scenario parameter set 507 to plan the trajectory for driving through that scenario. The speed planner 517 and... Figure 4 The speed decision module 405 performs the same function as the path planner 519. Figure 4 The path planning module 407 performs the same function. The updated scene parameter set 507 can be used as additional constraints when the planning module 305 generates the planned trajectory.

[0091] For example, the fixed scene parameters for an intersection might use a distance of 12 meters from the traffic lights as a boundary parameter. However, when an autonomous vehicle enters the intersection scene during peak hours, the planning module 305 can invoke the corresponding neural network model to adjust the default boundary parameter of 12 meters and one or more other predefined fixed parameters for the scene. Since it's peak hours, the autonomous vehicle can adjust the boundary parameter to 10 meters, meaning that the autonomous vehicle 101 will enter the intersection driving mode earlier to handle the complexities that arise due to increased traffic congestion at the intersection.

[0092] Therefore, the fixed scenario parameter set reflects how an ordinary person would drive through a scenario based on historical driving statistics, without considering the scenario location and the time of entry into the scenario; while the updated scenario parameter set reflects how a human driver would drive in a specific scenario based on the specific environmental conditions of the scenario and the status information of the vehicle in the scenario.

[0093] Figure 6 This illustrates how a fixed scene parameter 515 is generated according to one embodiment.

[0094] like Figure 6 As shown, in the planning module 305, multiple neural network models 613, 615, and 617 can be provided for different types of predefined scenarios. Each neural network model can be a multi-layer perceptron (MLP) and can have a different number of layers. However, each neural network model can take the same type of input, namely environmental data 601, 606, and 609 and vehicle state information 603, 607, and 611.

[0095] In one embodiment, each neural network model can be trained using driving statistics collected from various vehicles driven by human drivers. Each vehicle state information 603, 607, and 611 may include information indicating driving commands issued while driving through a corresponding scenario (e.g., accelerator, brake, steering commands) and vehicle responses (e.g., speed, acceleration, deceleration, direction) captured by vehicle sensors at different points in time. Each environmental data 601, 606, and 609 may include information describing the driving environment at different points in time, such as the route (including starting and destination locations), MPOI, road conditions, weather conditions, etc.

[0096] like Figure 6 As shown, the neural network model can generate a fixed set of scene parameters for each predefined scene 619, 621, and 623.

[0097] Figure 7 The figure illustrates a planning module 305 according to another embodiment. More specifically, the figure shows the process by which the planning module 305 generates an updated set of scene parameters in real time when the autonomous vehicle 101 enters a new scene.

[0098] In one embodiment, the planning module 305 may invoke the map-based scene inspector 702 (and...) in each planning cycle. Figure 5 The map-based scene inspector 501 (which is the same component) determines whether the autonomous vehicle 101 has entered a certain scene based on map information 701, route information 703, and vehicle status information (e.g., location) 705. The map-based scene inspector 702 can also make decisions based on historical environmental conditions 704 and historical vehicle status 706, using one or more scene parameters (e.g., scene boundaries) generated by corresponding neural network models in multiple models 711.

[0099] like Figure 7 As shown, the map-based scene inspector 702 can check whether the autonomous vehicle 101 has entered one of a plurality of predetermined scenes for the autonomous vehicle 101. Examples of predefined scenes may include intersections 713, yield signs 715, and many other scenes commonly found on maps, such as left turn, right turn, and stop signs. In addition, predefined scenes may include a default scene 716, which can represent any undefined scene for the autonomous vehicle 101.

[0100] Figure 7The diagram shows that the planning module 305 has determined that the autonomous vehicle 101 has entered the intersection scenario 713. At this time, the planning module 305 can call the neural network model 719, inputting the current vehicle state information and the current environmental conditions into the neural network model to generate an updated scenario parameter set. This updated scenario parameter set can partially or completely replace the fixed scenario parameter set generated for the intersection scenario 713, and the planning module 305 configures the autonomous vehicle 101 to drive in the intersection driving mode 723.

[0101] In one embodiment, an intersection driving mode 723 can be defined by a set of driving commands that reflect how a human driver would operate the autonomous vehicle 101 at an intersection 713 based on real-time environmental conditions 718 and real-time vehicle status (e.g., speed) 716 according to the intersection driving mode 723.

[0102] In one embodiment, if the planning module 305 cannot determine the scene type in which the autonomous vehicle 101 is located, the autonomous vehicle 101 will determine that it is driving in the default scene 716. Like other predefined scenes, the default scene 716 also has a fixed scene parameter set generated by a corresponding neural network model. This neural network model can also update the fixed scene parameter set according to real-time environmental conditions and vehicle status information, configuring the autonomous vehicle 101 to drive in the default driving mode 721.

[0103] In one embodiment, the neural network model used to generate fixed scene parameters and the neural network model used to generate updated scene parameters for each scene can be the same neural network model, or they can be different types of neural network models with different numbers of hidden layers.

[0104] Figure 8 This is a flowchart illustrating the operation process of an autonomous vehicle according to one embodiment. The process can be executed by processing logic, which may include software, hardware, or a combination thereof. For example, it can be performed via… Figure 5 The planning module 305 executes this process.

[0105] See Figure 8 In operation 801, the processing logic determines the scene that the autonomous vehicle has entered in the road segment. The autonomous vehicle can determine this based on its map information, route information, location information, and at least one boundary parameter from a fixed parameter set of the scene.

[0106] In operation 803, the processing logic invokes a trained neural network model corresponding to the scene to generate an updated scene parameter set based on the scene's environmental data and the autonomous vehicle's vehicle state information. The environmental data is real-time data related to the scene environment, and the autonomous vehicle's vehicle state information includes the autonomous vehicle's speed, heading, deceleration, acceleration, and various control commands.

[0107] In operation 805, the processing logic operates the autonomous vehicle, causing it to drive through the scenario in a driving mode defined by a scenario parameter set. The driving mode simulates the driving behavior of a human driver when navigating the scenario and is defined by an updated parameter set. For example, the updated parameter set for a right-turn scenario may include when the autonomous vehicle will decelerate to a certain speed, when it will resume cruising, when it will make a right turn, and the steering wheel angle.

[0108] It should be noted that some or all of the components shown and described above may be implemented by software, hardware, or a combination thereof. For example, these components may be implemented as software installed and stored in persistent storage, which may be loaded into memory and executed by a processor (not shown) to perform the processes or operations described herein. Alternatively, these components may be implemented as executable code programmed or embedded in dedicated hardware, such as integrated circuits (e.g., application-specific integrated circuits or ASICs), digital signal processors (DSPs), or field-programmable gate arrays (FPGAs), which may be accessed from an application via appropriate drivers and / or operating systems. Furthermore, these components may be implemented as specific hardware logic within a processor or processor core as part of an instruction set accessible by software components via one or more specific instructions.

[0109] Some parts of the foregoing detailed description have been introduced based on algorithms and symbolic representations for manipulating data bits in computer memory. These algorithmic descriptions and representations are the means by which those skilled in the art of data processing most effectively communicate the substance of their work to others skilled in the art. In this paper, algorithms are generally considered to be self-consistent sequences of operations that lead to desired results. These operations refer to those that require physical manipulation of physical quantities.

[0110] However, it should be remembered that all these terms and similar terms will be associated with appropriate physical quantities and are merely convenient notations applied to those quantities. Unless explicitly stated in the above discussion, it should be understood that throughout the specification, the use of terms such as those set forth in the appended claims refers to the operation and processing of a computer system or similar electronic computing device that manipulates data represented as physical (electronic) quantities in computer system registers and memories, and transforms that data into other data similarly represented as physical quantities in computer system memories or registers or other such information storage devices, transmission devices or display devices.

[0111] Embodiments of this disclosure also relate to means for performing the operations described herein. Such a computer program is stored in a non-transitory computer-readable medium. Machine-readable media include any mechanism for storing information in a machine-readable (e.g., computer-readable) form. For example, machine-readable (e.g., computer-readable) media include machine-readable (e.g., computer-readable) storage media (e.g., read-only memory (“ROM”), random access memory (“RAM”), disk storage media, optical storage media, flash memory devices).

[0112] The processes or methods depicted in the foregoing figures can be executed by processing logic, which includes hardware (e.g., circuits, special-purpose logic, etc.), software (e.g., embodied in a non-transitory computer-readable medium), or a combination of both. Although the process or method has been described above according to some sequential operations, it should be understood that some of the described operations can be performed in a different order. Furthermore, some operations can be performed in parallel rather than sequentially.

[0113] The embodiments disclosed herein are not described with reference to any particular programming language. It should be understood that the teachings of the embodiments of this disclosure as described herein can be implemented using a variety of programming languages.

[0114] In the foregoing description, embodiments of the present disclosure have been described with reference to specific exemplary embodiments. It is obvious that various modifications can be made to the present disclosure without departing from the broader spirit and scope of the disclosure as set forth in the appended claims. Therefore, the description and drawings should be regarded as illustrative rather than restrictive.

Claims

1. A computer-implemented method for operating an autonomous vehicle (ADV), comprising: The scene inspector on the autonomous vehicle determines the scene that the autonomous vehicle has entered in the road segment. The autonomous vehicle includes multiple fixed parameter sets corresponding to multiple predefined scenes. The fixed parameter sets are generated by a neural network model trained based on training data collected from manually driven vehicles that drive through the scene. Using the autonomous vehicle, a trained neural network model corresponding to the scenario is invoked. Based on environmental data of the scenario and the vehicle state information of the autonomous vehicle, the trained neural network model generates an updated parameter set for the scenario, wherein the updated parameter set replaces the fixed parameter set; and Operate the autonomous vehicle so that it travels through the scene in a driving mode defined by the updated parameter set; The scene inspector determines the scene that the autonomous vehicle has entered based on the current location of the autonomous vehicle, map information around the current location, information about the route the autonomous vehicle is taking, and one or more fixed parameters from a fixed parameter set of the scene.

2. The computer-implemented method according to claim 1, wherein, Based on historical environmental data and historical vehicle status information, a fixed set of parameters for the scenario is generated through the trained neural network model.

3. The computer-implemented method according to claim 1, wherein, The environmental data of the scenario includes the traffic volume of the scenario, and the vehicle status information of the autonomous vehicle includes the speed of the autonomous vehicle.

4. The computer-implemented method according to claim 1, wherein, The updated parameter set represents a set of additional constraints used by the autonomous vehicle when generating the planned trajectory.

5. The computer-implemented method according to claim 1, wherein, The scene inspector on the autonomous vehicle is a convolutional neural network (CNN).

6. The computer-implemented method according to claim 1, wherein, The scenario is one of several scenarios predefined based on map information. These scenarios include intersections, yield signs, stop signs, single-lane right turns, single-lane left turns, multi-lane right turns, and multi-lane left turns.

7. A non-transitory computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform the following operations for operating an autonomous driving vehicle (ADV): Determine the scenario in which the autonomous vehicle has entered the road segment, where, The autonomous vehicle includes multiple fixed parameter sets corresponding to multiple predefined scenarios. The fixed parameter sets are generated by a neural network model trained based on training data collected from manually driven vehicles that drive through the scenario. The trained neural network model corresponding to the scenario is invoked. Based on the environmental data of the scenario and the vehicle state information of the autonomous vehicle, the trained neural network model generates an updated parameter set for the scenario, wherein the updated parameter set replaces the fixed parameter set; and Operate the autonomous vehicle so that it travels through the scene in a driving mode defined by the updated parameter set; Specifically, the scene that the autonomous vehicle has entered is determined based on the current location of the autonomous vehicle, map information around the current location, information on the route that the autonomous vehicle is taking, and one or more fixed parameters from the fixed parameter set of the scene.

8. The non-transitory computer-readable medium according to claim 7, wherein, Based on historical environmental data and historical vehicle status information, a fixed set of parameters for the scenario is generated through the trained neural network model.

9. The non-transitory computer-readable medium according to claim 7, wherein, The environmental data of the scenario includes the traffic volume of the scenario, and the vehicle status information of the autonomous vehicle includes the speed of the autonomous vehicle.

10. The non-transitory computer-readable medium according to claim 7, wherein, The scenario parameter set represents a set of additional constraints used by the autonomous vehicle when generating the planned trajectory.

11. The non-transitory computer-readable medium according to claim 7, wherein, The scene inspector on the autonomous vehicle is a convolutional neural network (CNN).

12. The non-transitory computer-readable medium according to claim 7, wherein, The scenario is one of several scenarios predefined based on map information. These scenarios include intersections, yield signs, stop signs, single-lane right turns, single-lane left turns, multi-lane right turns, and multi-lane left turns.

13. An electronic device, comprising: processor; as well as A memory coupled to the processor stores instructions that, when executed by the processor, cause the processor to perform the following operations to operate the autonomous vehicle (ADV): The autonomous vehicle has entered a road segment, wherein the autonomous vehicle includes multiple fixed parameter sets corresponding to multiple predefined scenarios, and the fixed parameter sets are generated by a neural network model trained based on training data collected from human-driven vehicles that have driven through the scenario. The trained neural network model corresponding to the scenario is invoked. Based on the environmental data of the scenario and the vehicle state information of the autonomous vehicle, the trained neural network model generates an updated parameter set for the scenario, wherein the updated parameter set replaces the fixed parameter set; and Operate the autonomous vehicle so that it travels through the scene in a driving mode defined by the updated parameter set; Specifically, the scene that the autonomous vehicle has entered is determined based on the current location of the autonomous vehicle, map information around the current location, information on the route that the autonomous vehicle is taking, and one or more fixed parameters from the fixed parameter set of the scene.

14. The electronic device according to claim 13, wherein, Based on historical environmental data and historical vehicle status information, a fixed set of parameters for the scenario is generated through the trained neural network model.

15. The electronic device according to claim 13, wherein, The environmental data of the scenario includes the traffic volume of the scenario, and the vehicle status information of the autonomous vehicle includes the speed of the autonomous vehicle.

16. The electronic device according to claim 13, wherein, The scenario parameter set represents a set of additional constraints used by the autonomous vehicle when generating the planned trajectory.

17. The electronic device according to claim 13, wherein, The scene inspector on the autonomous vehicle is a convolutional neural network (CNN).

Images