Method and system for evaluating planning performance of autonomous vehicles

Abstract

A method and system for evaluating planning performance of an autonomous driving vehicle (ADV) are disclosed. The method includes the following operations: at an analysis application, receiving a log file recorded by the ADV for a driving scenario in a region, and a high-definition map matching the region; extracting planning messages and perception messages from the log file; and aligning the planning messages and the perception messages based on their timestamps. The method further includes calculating a single performance score for each planning cycle of the ADV for the driving scenario based on the planning messages; calculating a weight for each planning cycle based on the perception messages and the high-definition map; and then calculating a weighted score for the driving scenario based on the single performance scores and their corresponding weights.

Description

Technical Field

[0001] Embodiments of this disclosure generally relate to operating autonomous vehicles. More specifically, embodiments of this disclosure relate to evaluating the motion planning performance of autonomous vehicles. Background Technology

[0002] Vehicles operating in autonomous mode (e.g., driverless) can reduce some of the driving-related responsibilities for passengers, especially the driver. When operating in autonomous mode, the vehicle can use onboard sensors to navigate to various locations, allowing the vehicle to operate with minimal human-machine interaction or even without any passengers.

[0003] Autonomous vehicles rely on various modules for autonomous driving. One important module is the planning module, which generates trajectories for the vehicle to follow. Accurate evaluation of the planning module's performance is beneficial for vehicle calibration and adjustment.

[0004] However, when evaluating the motion planning performance of autonomous vehicles in scenarios such as navigating traffic intersections, existing solutions typically examine only a single planned trajectory based on fields such as acceleration. If a single score for the scenario is required, a simple average score across all frames is used to indicate the performance of the planning module throughout the scenario. These existing solutions do not consider the different traffic conditions at different stages of the driving scenario, and therefore cannot accurately evaluate the vehicle's motion planning performance. Summary of the Invention

[0005] On one hand, a computer-based method for evaluating the planning performance of an autonomous driving vehicle (ADV), the method comprising:

[0006] Through an analysis application executed by the processor, multiple planning messages and multiple perception messages are extracted from a log file containing previous driving records of ADVs associated with driving scenarios in the area.

[0007] By analyzing the application, a single performance score is calculated for each of the multiple planning cycles of ADV for a driving scenario, based on multiple planning messages.

[0008] By analyzing the application, the weights for each of multiple planning cycles are calculated based on perceived messages and region-related maps; and

[0009] By analyzing the application, a weighted score for the driving scenario is calculated based on the individual performance score and corresponding weight of each of the multiple planning cycles.

[0010] On the other hand, a non-transitory machine-readable medium storing instructions is provided, which, when executed by a processor, causes the processor to perform a computer implementation method for evaluating the planning performance of an autonomous driving vehicle (ADV) as described above.

[0011] On the other hand, a data processing system is provided, including:

[0012] Processor; and

[0013] The memory, coupled to the processor, stores instructions that, when executed by the processor, cause the processor to perform operations as described in the computer implementation of the method for evaluating the planning performance of an autonomous driving vehicle (ADV).

[0014] This disclosure enables a more accurate assessment of the motion planning performance of autonomous vehicles. Attached Figure Description

[0015] Embodiments of this disclosure are shown by way of example and are not limited to the figures in the accompanying drawings, in which the same reference numerals denote similar elements.

[0016] Figure 1 This is a block diagram illustrating a networking system according to one embodiment.

[0017] Figure 2 This is a block diagram illustrating an example of an autonomous vehicle according to one embodiment.

[0018] Figures 3A-3B This is a block diagram illustrating an example of a perception and planning system used with an autonomous vehicle according to one embodiment.

[0019] Figure 4 A system for evaluating the performance of an ADV planning module is shown according to an embodiment.

[0020] Figure 5 The factors used to calculate the weight of a frame according to an embodiment are shown.

[0021] Figure 6 This is a flowchart illustrating a process 600 for evaluating the performance of the planning module of ADV according to an embodiment. Detailed Implementation

[0022] Various embodiments and aspects of this disclosure will be described with reference to the details of the following discussion, and the accompanying drawings will illustrate various embodiments. The following description and drawings are illustrative of this disclosure and should not be construed as limiting it. Numerous specific details are described to provide a full understanding of the various embodiments of this disclosure. However, in some cases, well-known or conventional details have not been described in order to provide a brief discussion of embodiments of this disclosure.

[0023] References to "an embodiment" or "embodiment" in the specification mean that a particular feature, structure, or characteristic described in connection with that embodiment may be included in at least one embodiment of this disclosure. The phrase "in an embodiment" appearing in various places in the specification does not necessarily refer to the same embodiment.

[0024] According to various embodiments, the systems disclosed herein are methods, systems, and media for evaluating the motion planning performance of autonomous vehicles navigating driving scenarios. Embodiments apply different weights to each stage of a driving scenario based on the complexity of the driving environment at each stage, thereby providing a more accurate assessment of the motion planning performance of the autonomous vehicle.

[0025] In one embodiment, an exemplary method includes the following operations: at an analytics application, receiving a log file of a driving scenario in a region recorded by an ADV, and a high-resolution map matching the region; extracting planning messages and perception messages from the log file; and aligning the planning messages and perception messages based on their timestamps. The method further includes calculating a single performance score for each planning cycle of the driving scenario for the ADV based on the planning messages; calculating a weight for each planning cycle based on the perception messages and the high-resolution map; and then calculating a weighted score for the driving scenario based on the single performance score and its corresponding weight.

[0026] In one embodiment, the planning period, also known as a frame, is a time interval (e.g., 100ms) during which a planned trajectory is generated for a subsequent time period (e.g., the next 2 seconds). The analytics application has one or more standard interfaces that allow users to upload log files and high-resolution maps, and the analytics application is configured to run on an ADV or cloud server.

[0027] In one embodiment, the weight for each planning cycle can be calculated based on one or more factors, including the curvature of the planned path, the number of obstacles with overlapping trajectories with the ADV, or the size of intersections. Each factor can be derived from perception messages or high-definition maps.

[0028] In one embodiment, calculating the weighted score for a driving scenario further includes the following operations: calculating a weighted score for each planning period by multiplying a single performance score of the planning period by the corresponding weight of the planning period; summing the weighted scores of each planning period to generate a total weighted score; and dividing the total weighted score by the number of planning periods.

[0029] In one embodiment, the driving scenario comprises multiple stages, each corresponding to one or more planning cycles and having a different driving environment. A single performance score for each planning cycle is calculated based on factors measuring controllability and comfort.

[0030] The above embodiments are not exhaustive of all aspects of the invention. The invention is intended to encompass all suitable combinations of the various embodiments summarized above, as well as all embodiments implemented in those disclosed below.

[0031] autonomous vehicles

[0032] Figure 1 This is a block diagram illustrating an autonomous driving network configuration according to an embodiment of the present disclosure. (See reference...) Figure 1 Network configuration 100 includes an autonomous vehicle (ADV) 101, which can be communicatively coupled to one or more servers 103-104 via network 102. Although only one ADV is shown, multiple ADVs can be coupled to each other and / or to servers 103-104 via network 102. Network 102 can be any type of network, such as a local area network (LAN), a wide area network (WAN) such as the Internet, a cellular network, a satellite network, or a combination thereof, wired or wireless. Servers 103-104 can be any type of server or server cluster, such as a web or cloud server, an application server, a backend server, or a combination thereof. Servers 103-104 can be data analytics servers, content servers, traffic information servers, map and point of interest (MPOI) servers, or location servers, etc.

[0033] ADV refers to a vehicle that can be configured to operate in an autonomous mode, in which the vehicle navigates its environment with little or no driver input. Such an ADV may include a sensor system with one or more sensors configured to detect information about the environment in which the vehicle operates. The vehicle and its associated controller use the detected information to navigate through the environment. ADV 101 can operate in manual mode, fully autonomous mode, or partially autonomous mode.

[0034] In one embodiment, ADV 101 includes, but is not limited to, an autonomous driving system (ADS) 110, a vehicle control system 111, a wireless communication system 112, a user interface system 113, and a sensor system 115. ADV 101 may also include certain common components found in ordinary vehicles, such as an engine, wheels, steering wheel, transmission, etc., which can be controlled by the vehicle control system 111 and / or ADS 110 using various communication signals and / or commands (e.g., acceleration signals or commands, deceleration signals or commands, steering signals or commands, braking signals or commands, etc.).

[0035] Components 110-115 can be communicatively coupled to each other via interconnect, bus, network, or a combination thereof. For example, components 110-115 can be communicatively coupled to each other via a Controller Area Network (CAN) bus. The CAN bus is a vehicle bus standard designed to allow microcontrollers and devices to communicate with each other in masterless applications. It is a message-based protocol originally designed for multiplexing electrical wiring in automobiles, but is also used in many other environments.

[0036] Now for reference Figure 2 In one embodiment, the sensor system 115 includes, but is not limited to, one or more cameras 211, a Global Positioning System (GPS) unit 212, an Inertial Measurement Unit (IMU) 213, a radar unit 214, and a light detection and range (LIDAR) unit 215. The GPS system 212 may include a transceiver operable to provide information about the ADV's location. The IMU unit 213 may sense changes in the ADV's position and orientation based on inertial acceleration. The radar unit 214 may represent a system that uses radio signals to sense objects within the ADV's local environment. In some embodiments, in addition to sensing objects, the radar unit 214 may additionally sense the velocity and / or heading of objects. The LIDAR unit 215 may use lasers to sense objects in the ADV's environment. The LIDAR unit 215 may include one or more laser sources, a laser scanner, and one or more detectors, as well as other system components. The camera 211 may include one or more devices to capture images of the environment surrounding the ADV. The camera 211 may be a still camera and / or a video camera. The camera may be mechanically movable, for example, by mounting the camera on a rotating and / or tilting platform.

[0037] The sensor system 115 may also include other sensors, such as sonar sensors, infrared sensors, steering sensors, throttle sensors, brake sensors, and audio sensors (e.g., microphones). The audio sensor can be configured to capture sound from the environment surrounding the ADV. The steering sensor can be configured to sense the steering angle of the steering wheel, the vehicle's wheels, or a combination thereof. The throttle and brake sensors sense the vehicle's throttle and brake positions, respectively. In some cases, the throttle and brake sensors can be integrated into an integrated throttle / brake sensor.

[0038] In one embodiment, the vehicle control system 111 includes, but is not limited to, a steering unit 201, a throttle unit 202 (also referred to as an acceleration unit), and a braking unit 203. The steering unit 201 is used to adjust the direction or heading of the vehicle. The throttle unit 202 is used to control the speed of a motor or engine, which in turn controls the speed and acceleration of the vehicle. The braking unit 203 decelerates the vehicle by providing friction to slow down the wheels or tires. Note that... Figure 2 The components shown can be implemented in hardware, software, or a combination thereof.

[0039] Return to reference Figure 1 The wireless communication system 112 allows communication between ADV 101 and external systems, such as devices, sensors, other vehicles, etc. For example, the wireless communication system 112 can communicate wirelessly with one or more devices directly or via a communication network, such as communicating with servers 103-104 via network 102. The wireless communication system 112 can use any cellular communication network or wireless local area network (WLAN), such as using WiFi, to communicate with another component or system. The wireless communication system 112 can communicate directly with devices (e.g., passenger mobile devices, display devices, speakers within vehicle 101), for example, using infrared links, Bluetooth, etc. The user interface system 113 can be part of peripheral devices implemented within vehicle 101, including, for example, a keyboard, touchscreen display, microphone, and speakers.

[0040] Some or all of the functions of ADV 101 can be controlled or managed by ADS 110, especially when operating in autonomous driving mode. ADS 110 includes the necessary hardware (e.g., processor, memory, storage devices) and software (e.g., operating system, planning and routing programs) to receive information from sensor system 115, control system 111, wireless communication system 112, and / or user interface system 113, process the received information, plan a route or path from the origin to the destination, and then drive vehicle 101 based on the planning and control information. Alternatively, ADS 110 can be integrated with vehicle control system 111.

[0041] For example, a passenger can specify the start and destination of their trip via a user interface. The ADS 110 obtains trip-related data. For instance, the ADS 110 can obtain location and route information from an MPOI server, which may be part of servers 103-104. The location server provides location services, and the MPOI server provides map services and points of interest (POIs) for certain locations. Alternatively, this location and MPOI information can be cached locally in the ADS 110's persistent storage.

[0042] As ADV 101 moves along the route, ADS 110 can also obtain real-time traffic information from a traffic information system or server (TIS). Note that servers 103-104 can be operated by a third-party entity. Alternatively, the functionality of servers 103-104 can be integrated with ADS 110. Based on real-time traffic information, MPOI information, location information, and real-time local environmental data (e.g., obstacles, objects, nearby vehicles) detected or sensed by sensor system 115, ADS 110 can plan an optimal route and, for example, drive vehicle 101 according to the planned route via control system 111 to safely and efficiently reach the designated destination.

[0043] Server 103 may be a data analysis system that performs data analysis services for various clients. In one embodiment, data analysis system 103 includes a data collector 121 and a machine learning engine 122. Data collector 121 collects driving statistics 123 from various vehicles (ADVs or conventional vehicles driven by human drivers). Driving statistics 123 include information indicating issued driving commands (e.g., accelerator, brake, steering commands) and vehicle responses (e.g., speed, acceleration, deceleration, direction) captured by vehicle sensors at different points in time. Driving statistics 123 may also include information describing the driving environment at different points in time, such as route (including starting and destination locations), MPOI, road conditions, weather conditions, etc.

[0044] Based on driving statistics 123, machine learning engine 122 generates or trains a set of rules, algorithms, and / or prediction models 124 for various purposes. Algorithm 124 can then be uploaded to ADV for use in real time during autonomous driving.

[0045] Server 103 may also include a motion planning performance analysis application 126, which may be a cloud service, to evaluate the performance of the motion planning of ADV, such as the performance of the planning module as described below.

[0046] Figure 3A and 3B This is a block diagram illustrating an example of an autonomous driving system used with an ADV according to one embodiment. System 300 can be implemented as follows: Figure 1 This is part of ADV 101, including but not limited to ADS 110, control system 111, and sensor system 115. (See reference) Figures 3A-3B The ADS 110 includes, but is not limited to, a positioning module 301, a perception module 302, a prediction module 303, a decision-making module 304, a planning module 305, a control module 306, a routing module 307, and a driving recorder 308.

[0047] Some or all of modules 301-308 may be implemented in software, hardware, or a combination thereof. For example, these modules may be installed in permanent storage device 352, loaded into memory 351, and executed by one or more processors (not shown). Note that some or all of these modules may be compatible with… Figure 2 Some or all modules of the vehicle control system 111 are communicatively coupled or integrated. Some of modules 301-308 can be integrated together as integrated modules.

[0048] The positioning module 301 determines the current location of the ADV 300 (e.g., using GPS unit 212) and manages any data related to the user's trip or route. The positioning module 301 (also called the map and route module) manages any data related to the user's trip or route. The user can log in, for example, via a user interface and specify the start and destination of the trip. The positioning module 301 communicates with other components of the ADV 300, such as map and route data 311, to obtain trip-related data. For example, the positioning module 301 can obtain location and route data from a location server and a map and POI (MPOI) server. The location server provides location services, and the MPOI server provides map services and POIs for certain locations, which can be cached as part of the map and route data 311. As the ADV 300 moves along the route, the positioning module 301 can also obtain real-time traffic information from a traffic information system or server.

[0049] Based on sensor data provided by sensor system 115 and positioning information obtained by positioning module 301, perception module 302 determines the perception of the surrounding environment. The perception information can represent the situation around the vehicle being driven by a typical driver. Perception may include, for example, the relative positions of lane configurations, traffic light signals, other vehicles, pedestrians, buildings, crosswalks, or other traffic-related signs (e.g., stop signs, yield signs), etc. Lane configurations include information describing one or more lanes, such as, for example, the shape of the lanes (e.g., straight or curved), the width of the lanes, the number of lanes in the road, one-way or two-way lanes, merging or separating lanes, exiting lanes, etc.

[0050] The perception module 302 may include a computer vision system or the functionality of a computer vision system to process and analyze images captured by one or more cameras to identify objects and / or features in the ADV's environment. Objects may include traffic signals, lane boundaries, other vehicles, pedestrians and / or obstacles, etc. The computer vision system may use object recognition algorithms, video tracking, and other computer vision techniques. In some embodiments, the computer vision system may map the environment, track objects, and estimate the velocity of objects, etc. The perception module 302 may also detect objects based on additional sensor data provided by other sensors such as radar and / or LIDAR.

[0051] The perception module 302 may include an emergency vehicle detection module 308, which can use both audio and visual data to detect the presence of emergency vehicles in the ADV’s surrounding environment.

[0052] For each object, prediction module 303 predicts how the object will behave in the environment. Based on a set of map / route information 311 and traffic rules 312, predictions are performed using perceived data of the driving environment at the point of perception. For example, if the object is a vehicle traveling in the opposite direction and the current driving environment includes an intersection, prediction module 303 will predict whether the vehicle is likely to move straight ahead or turn. If the perceived data indicates that there are no traffic lights at the intersection, prediction module 303 can predict that the vehicle may have to come to a complete stop before entering the intersection. If the perceived data indicates that the vehicle is currently in a left-turn-only lane or a right-turn-only lane, prediction module 303 can predict that the vehicle is more likely to make a left turn or a right turn, respectively.

[0053] For each object, decision module 304 makes a decision about how to handle that object. For example, given a specific object (e.g., another vehicle at an intersection) and metadata describing that object (e.g., speed, direction, steering angle), decision module 304 decides how to encounter the object (e.g., overtake, yield, stop, pass). Decision module 304 may make these decisions based on a set of rules, such as traffic rules or driving rules 312, which may be stored in permanent storage device 352.

[0054] The routing module 307 is configured to provide one or more routes or paths from the origin to the destination. For a given trip from the origin to the destination received from the user, for example, the routing module 307 obtains route and map information 311 and determines all possible routes or paths from the origin to the destination. The routing module 307 can generate reference lines in the form of topographic maps for each route it determines from the origin to the destination. The reference lines refer to ideal routes or paths free from any interference from other vehicles, obstacles, or traffic conditions. That is, if there are no other vehicles, pedestrians, or obstacles on the road, the ADV should follow the reference lines precisely or closely. The topographic map is then provided to the decision module 304 and / or the planning module 305. The decision module 304 and / or the planning module 305 examine all possible routes to select and refine one of the optimal routes based on other data provided by other modules (such as traffic conditions from the positioning module 301, the driving environment perceived by the perception module 302, and traffic conditions predicted by the prediction module 303). Depending on the specific driving conditions at a given point in time, the actual path or route used to control the ADV may be close to or different from the reference line provided by the routing module 307.

[0055] Based on the decision for each perceived object, the planning module 305 uses reference lines provided by the routing module 307 as a basis to plan a path, route, or trajectory for ADV, along with driving parameters (e.g., distance, speed, and / or steering angle). That is, for a given object, the decision module 304 decides what to do with that object, while the planning module 305 determines how to do it. For example, for a given object, the decision module 304 might decide to pass the object, while the planning module 305 might determine whether to pass to the left or right of the object. Planning and control data is generated by the planning module 305 and includes information describing how the vehicle 300 will move in the next movement cycle (e.g., the next route / path segment). For example, the planning and control data might instruct the vehicle 300 to move 10 meters at 30 miles per hour (mph) and then change lanes to the right at 25 mph.

[0056] Based on planning and control data, control module 306 controls and drives the ADV by sending appropriate commands or signals to vehicle control system 111 via CAN bus module 321 according to the trajectory (also known as route or path) defined by the planning and control data. The planning and control data includes sufficient information to drive the vehicle from one point to another along the route or path at different times using appropriate vehicle settings or driving parameters (e.g., throttle, braking, steering commands).

[0057] In one embodiment, the planning phase is performed in multiple planning cycles, also known as driving cycles, such as, for example, in each time interval of 100 milliseconds (ms). For each planning cycle or driving cycle, one or more control commands are issued based on the planning and control data. That is, for every 100 ms, the planning module 305 plans the next route segment or path segment, including, for example, the target location and the time required for the ADV to reach the target location. Alternatively, the planning module 305 may further specify a particular speed, direction, and / or steering angle, etc. In one embodiment, the planning module 305 plans a route segment or path segment for the next predetermined time period, such as 5 seconds. For each planning cycle, the planning module 305 plans a target location for the current cycle (e.g., the next 5 seconds) based on the target location planned in the previous cycle. The control module 306 then generates one or more control commands (e.g., throttle, braking, steering control commands) based on the planning and control data of the current cycle.

[0058] Note that the decision module 304 and the planning module 305 can be integrated into an integrated module. The decision module 304 / planning module 305 may include a navigation system or the functionality of a navigation system to determine a driving path for the ADV. For example, the navigation system may determine a series of speeds and headings to influence the movement of the ADV along a path that substantially avoids perceived obstacles, while generally guiding the ADV along a road-based path leading to the final destination. The destination can be set based on user input via the user interface system 113. The navigation system can dynamically update the driving path while the ADV is in operation. The navigation system may incorporate data from a GPS system and one or more maps to determine the driving path for the ADV 101.

[0059] Driving recorder 309 records driving data from at least three data channels of the ADV control system: a control channel, a chassis channel, and a positioning channel. The control channel generates information about control commands to the ADV's control systems, such as braking, throttle, and steering. The chassis channel generates information from various sensors, such as accelerometers, and readings of the actual position or actuation of the braking, throttle, and steering systems. The positioning channel references standard references such as high-definition (HD) maps or Global Positioning Satellite (GPS) systems to generate information about the ADV's actual position and heading. Driving data can be recorded at approximately 100 frames per second (fps) or approximately 10 milliseconds (ms) per frame. Each driving record has a timestamp. The timestamp can be an absolute timestamp in the form of hh:mm:ss:ms (hours, minutes, seconds, milliseconds) relative to a start time, such as the start of a driving route. In an embodiment, the timestamp can be a frame number relative to a start time (such as the start of a driving route). In an embodiment, each driving record may also have a datestamp in addition to the timestamp. The driving recorder 309 can record driving data for both simulated ADV driving sessions and real-world ADV driving sessions.

[0060] The driving recorder 309 can write driving records to a non-volatile storage device, such as driving log storage device 313. The driving log 313 can be uploaded autonomously or manually to a server system, such as servers 103-104, to generate a set of standardized performance metrics to classify the performance of one of the multiple autonomous driving modules of the ADV.

[0061] Motion Planning Performance Analysis

[0062] Figure 4 A system for evaluating the performance of an ADV planning module is shown according to an embodiment.

[0063] Motion planning performance analysis application 126 can run on a server, such as Figure 1 The server 103 described herein; and can receive recording files recorded by a driving recorder (e.g., driving recorder 308) directly from the autonomous vehicle during road testing or simulation. The motion planning performance analysis application 126 may also include multiple standard interfaces allowing users to upload recording files 401 to server 103 and, during simulation or road testing, specify a high-resolution map 411 matching the area the ADV is traveling on. In one embodiment, the recording files are in a specific format, such as Baidu's Apollo file format.

[0064] As further shown, the motion planning performance analysis application 126 can extract planning messages 403 and perception messages 405 from the log file 401. The planning messages 403 and perception messages 405 can be aligned according to their timestamps.

[0065] As used herein, messages are the real-time output of the corresponding module while that module is operating the autonomous vehicle. For example, the planning module may generate a planning message for each frame (e.g., every 100 milliseconds) and may provide a planned path for subsequent time intervals (e.g., the next 2 seconds). The planning message may provide information about reference points at uniform distances from each other on the planned path / trajectory generated by the ADV for each frame. For example, the planning message may specify the ADV's expected orientation, velocity, and curvature at each reference point. Perception messages may be generated by the perception module and may include information collected by cameras and / or LiDAR devices, such as lane configurations, traffic light signals, the relative position of other vehicles, pedestrians, buildings, crosswalks, or other traffic-related signs (e.g., stop signs, yield signs), etc.

[0066] A single trajectory evaluator 407 can receive planning messages 403, evaluate the planning messages 403 based on factors such as controllability and comfort, and generate a single score indicating the performance of the planning module in each planning cycle, i.e., frame. In this invention, controllability can be measured by factors such as non-shift trajectory length ratio, initial heading difference ratio, normalized curvature ratio, rate of change of curvature ratio, acceleration ratio, deceleration ratio, and longitudinal jerk ratio. Comfort can be measured by factors such as longitudinal jerk ratio, lateral jerk ratio, longitudinal and lateral acceleration ratios, longitudinal deceleration ratio, lateral deceleration ratio, boundary distance ratio, obstacle distance ratio, and collision time ratio.

[0067] For a specific driving scenario with a particular duration (e.g., from 20 seconds to 5 minutes), the ADV may experience multiple frames. In one example, a frame lasts for 100 ms. During each frame, the planning module can generate a planned trajectory for the ADV in the subsequent time interval (e.g., the next 2 seconds). For each frame, a single trajectory evaluator 407 can calculate a single performance score based on factors of controllability and comfort.

[0068] While calculating a single performance score for a frame, the traffic complexity estimator 409 can calculate the frame's weight based on information from the high-definition map 411 and perception messages 405. Information from the high-definition map may include traffic signs and lane markings. Perception messages 405 can provide traffic conditions such as those of surrounding vehicles and pedestrians. The following... Figure 5 The method for calculating the weight of each frame is described in the document.

[0069] After obtaining a single performance score based on the planning message 403 for each frame of the driving scenario (e.g., passing through a red light at an intersection) and the weights for that frame, the motion planning performance analysis application 126 can calculate a final weighted score 413 for the entire driving scenario based on the formula SUM(score * weights) / total number of trajectories. Note that the total number of trajectories equals the total number of frames, because ADV generates one trajectory for each frame.

[0070] Figure 5 The factors used to calculate the weight of a frame according to an embodiment are shown.

[0071] As described above, embodiments of this disclosure consider the weights of different frames when calculating a scenario score to measure the performance of the planning module of an ADV (Advanced Driver Module). The frame weights reflect the complexity of road and traffic conditions when generating the planned trajectory. In driving scenarios, such as when an autonomous vehicle approaches a traffic intersection, the vehicle may go through different phases: cruising along the lane, coming to a complete stop when a red light is detected, accelerating when the traffic light turns green, and safely crossing the intersection. Each phase may differ in terms of the complexity of traffic and road conditions, thus posing different challenges to the planning module. Therefore, when calculating the final score to measure the performance of the planning module, the planned trajectory in different phases should not be given equal weight.

[0072] For example, following traffic flow along lanes is easier, so the planned trajectory in this stage can be assigned a lower weight. On the other hand, the planned trajectory involved in crossing intersections is more complex and should have a higher weight. Compared to uniform weighting methods, weighted analysis methods can better indicate the performance of the planning module.

[0073] In an embodiment, each stage may correspond to one or more frames of the autonomous vehicle, as it may cause the vehicle to spend more than one planning cycle (i.e., more than one frame) to pass through a stage. When a stage corresponds to more than one frame, each frame corresponding to that stage may have a higher weight.

[0074] Figure 5 The factors used to calculate the weights of a frame are shown. It can be deduced that the values ​​of factors 501-511, with weight 515, used to calculate the environment score are the values ​​of the factors during that frame.

[0075] like Figure 5As shown, such factors may include path curvature 507, the number of interactive obstacles 509 that have overlapping trajectories with the vehicle, and / or intersection size 511. After determining that the vehicle is at an intersection or roundabout based on traffic information 505 shown by perception message 405, path curvature 507 can be derived from the planned path 501 generated for the frame by the planning module, interactive obstacles 509 can be derived from obstacles 503 predicted during the frame based on perception message, and intersection size 511 can be derived from the corresponding portion of the high-definition map.

[0076] In one embodiment, a predetermined algorithm can be used to calculate an environment score 513 based on the values ​​of the aforementioned factors from the frame. From the environment score 513, a weight 515 for the frame can be derived. The weight 515 can be a fraction or a decimal and can be linearly proportional to the environment score.

[0077] Figure 6 This is a flowchart illustrating a process 600 for evaluating the performance of an ADV planning module according to an embodiment. Process 600 can be executed by processing logic that may include software, hardware, or a combination thereof. For example, process 600 can be performed by… Figure 1 and Figure 4 The motion planning performance analysis described in the document is performed using application 126.

[0078] like Figure 6 As shown, in operation 601, the processing logic receives a log file of a driving scenario in a region, recorded by the ADV, along with a high-resolution map matching the region. The driving scenario can last for a period of time and may include multiple phases, each with a different driving environment, such as different traffic conditions and / or road conditions. For example, when the autonomous vehicle approaches a traffic intersection, it may involve phases such as lane cruising, coming to a complete stop when a red light is detected, accelerating when the traffic light turns green, and safely navigating through the intersection. The complexity of the traffic situation varies for each phase, and therefore the difficulty of motion planning also varies. In operation 602, the processing logic extracts planning messages and perception messages from the log file. The log file is recorded by the ADV during simulation or road testing. In operation 603, the processing logic aligns the planning messages and perception messages based on their timestamps. In this operation, each planning message and each perception message is matched such that both messages cover the same planning cycle. In operation 604, the processing logic calculates a single performance score for each planning cycle of the ADV as it drives through the driving scenario. The single performance score is based on factors extracted from the planning messages. In operation 605, the processing logic calculates the weights for each planning cycle based on the perceived messages and the high-definition map. In operation 606, the processing logic calculates a weighted score for the driving scenario based on the individual performance score and corresponding weight for each planning cycle.

[0079] Note that some or all of the components shown and described above can be implemented in software, hardware, or a combination thereof. For example, these components can be implemented as software installed and stored in a permanent storage device, which can be loaded and executed in memory by a processor (not shown) to perform the processes or operations described throughout this application. Alternatively, these components can be implemented as executable code programmed or embedded in dedicated hardware such as integrated circuits (e.g., application-specific ICs or ASICs), digital signal processors (DSPs), or field-programmable gate arrays (FPGAs), accessible via corresponding drivers and / or operating systems from the application. Furthermore, these components can be implemented as specific hardware logic within a processor or processor core as part of an instruction set accessible via one or more specific instruction software components.

[0080] Some parts of the foregoing detailed description of algorithms and symbolic representations for operations on data bits within computer memory have already been presented. These algorithmic descriptions and representations are the most efficient way for those skilled in the art of data processing to communicate the substance of their work to others skilled in the art. Algorithms here and generally are considered to be self-consistent sequences of operations that lead to desired results. These operations are those that require physical manipulation of physical quantities.

[0081] However, it should be remembered that all these and similar terms are associated with appropriate physical quantities and are merely convenient notations applied to those quantities. Unless otherwise stated, it is obvious from the above discussion that it should be understood that throughout the specification, the use of terms such as those set forth in the appended claims refers to the actions and processes of a computer system or similar electronic computing device that manipulates and transforms data represented as physical (electronic) quantities in the registers and memories of the computer system into other data similarly represented as physical quantities in the computer system's memory or registers or other such information storage, transmission, or display devices.

[0082] Embodiments of this disclosure also relate to means for performing the operations described herein. Such a computer program is stored in a non-transitory computer-readable medium. Machine-readable media include any mechanism for storing information in a machine-readable (e.g., computer-readable) form. For example, machine-readable (e.g., computer-readable) media include machine-readable (e.g., computer-readable) storage media (e.g., read-only memory (“ROM”), random access memory (“RAM”), disk storage media, optical storage media, flash memory devices).

[0083] The processes or methods described in the foregoing figures can be performed by processing logic including hardware (e.g., circuitry, dedicated logic, etc.), software (e.g., embodied on a non-transitory computer-readable medium), or a combination of both. Although the processes or methods have been described above according to some sequential operations, it should be understood that some of the operations can be performed in different orders. Furthermore, some operations can be performed in parallel rather than sequentially.

[0084] The embodiments disclosed herein are not described with reference to any particular programming language. It will be understood that the teachings of the embodiments of this disclosure as described herein can be implemented using various programming languages.

[0085] In the foregoing description, embodiments of the present disclosure have been described with reference to specific exemplary embodiments. It will be apparent that various modifications may be made thereto without departing from the broader spirit and scope of the present disclosure as set forth in the appended claims. Therefore, the description and drawings should be considered illustrative rather than restrictive.

Claims

1. A computer-based method for evaluating the planning performance of an autonomous vehicle (ADV), the method comprising: Divide driving scenarios with a certain duration into multiple planning cycles; For each of the multiple planning cycles, one or more control commands are generated based on the planning and control data generated by the planning module for each planning cycle; Drive the ADV within a certain area using one or more control commands; Store ADV driving records related to ADV driving in driving scenarios within this area; Through an analysis application executed by the processor, multiple planning messages and multiple perception messages are extracted from a log file containing previous driving records of an ADV associated with driving scenarios in the area, and the multiple planning messages and multiple perception messages are aligned based on timestamps. The multiple planning messages are generated by the planning module, specifying the expected direction, speed and curvature of the ADV at each reference point, and the multiple perception messages are generated by the perception module based on information collected by cameras and / or LiDAR devices. By analyzing the application, a single performance score is calculated for each of the multiple planning cycles of ADV for a driving scenario, based on multiple planning messages. By analyzing the application, the weight of each of the multiple planning cycles is calculated based on one or more factors, such as the curvature of the planned path, the number of obstacles with overlapping trajectories with the ADV, or the intersection size. Each of the one or more factors is extracted from multiple perception messages or maps associated with the region. as well as By analyzing the application, a weighted score for the driving scenario is calculated based on the individual performance score and corresponding weight of each of the multiple planning cycles.

2. The method as described in claim 1, wherein, The analytics application is configured to run on ADV or a cloud server.

3. The method as described in claim 1, wherein, Calculating the weighted score for the driving scenario further includes: The weighted score for each of the multiple planning periods is calculated by multiplying the individual performance score of the planning period by the corresponding weight used for the planning period. The weighted scores for each planning period are summed to generate the total weighted score; and Divide the total weighted score by the number of planning periods.

4. The method of claim 1, wherein, The driving scenario includes multiple different stages, each corresponding to a planning cycle and having a different driving environment.

5. The method of claim 1, wherein, Individual performance scores for each of the multiple planning cycles are based on factors that measure controllability and comfort.

6. The method of claim 1, further comprising providing one or more interfaces that enable clients to upload log files and maps to determine the planning performance of ADV.

7. A non-transitory machine-readable medium storing instructions which, when executed by a processor, cause the processor to perform a computer-implemented method for evaluating the planning performance of an autonomous driving vehicle (ADV) as claimed in any one of claims 1 to 6.

8. A data processing system, comprising: processor; as well as The memory is coupled to the processor and stores instructions that, when executed by the processor, cause the processor to perform the operations of the computer implementation method for evaluating the planning performance of an autonomous driving vehicle (ADV) as described in any one of claims 1 to 6.

9. A computer program product comprising a computer program that, when executed by a processor, causes the processor to perform the method as described in any one of claims 1 to 6.

Images