Methods, media, and systems for evaluating an open space planner of an autonomous vehicle

Abstract

The present disclosure relates to methods, non-transitory machine-readable media, data processing systems, and computer program products for evaluating an open space planner of an autonomous driving vehicle (ADV). In one embodiment, an example method includes receiving, at an analysis application, a log file recorded by an ADV while driving in an open space using an open space planner and a configuration file specifying parameters of the ADV; extracting, from the log file, planning messages and prediction messages, each extracted message being associated with the open space planner. The method further includes generating features from the planning messages and the prediction messages in accordance with the specified parameters of the ADV; and computing statistical metrics from the features. The statistical metrics are then provided to an auto-tuning framework to tune the open space planner.

Description

Technical Field

[0001] The embodiments of this disclosure generally relate to operating autonomous vehicles. More specifically, embodiments of this disclosure relate to evaluating and adapting open space planners for autonomous vehicles. Background Technology

[0002] Vehicles operating in autonomous mode (e.g., driverless) can reduce some of the driving-related responsibilities of occupants, especially the driver. When operating in autonomous mode, the vehicle can use onboard sensors to navigate to various locations, allowing the vehicle to travel with minimal human-machine interaction or in certain situations without any passengers.

[0003] Trajectories are typically planned based on lanes pre-marked within high-definition (HD) maps. However, this process is unsuitable for certain scenarios, such as open spaces without available lanes, like parking lots. In such applications, an open space planner can be activated to operate the autonomous vehicle. The open space planner can be a standalone autonomous driving module within the vehicle and requires careful tuning before deployment. Therefore, the open space planner needs to be evaluated, and the evaluation results can be used for automatic tuning of the planner. Attached Figure Description

[0004] Embodiments of this disclosure are illustrated in the accompanying drawings by way of example rather than limitation, wherein similar reference numerals indicate similar elements.

[0005] Figure 1 This is a block diagram illustrating a networking system according to one embodiment.

[0006] Figure 2 This is a block diagram illustrating an example of an autonomous vehicle according to one embodiment.

[0007] Figures 3A to 3B This is a block diagram illustrating an example of a perception and planning system used with an autonomous vehicle according to one embodiment.

[0008] Figure 4 This is a block diagram illustrating an example of an open space planning module according to one embodiment.

[0009] Figure 5 This is a high-level flowchart illustrating the process of adjusting the parameters of an open space planner according to an embodiment.

[0010] Figure 6 This is a system for evaluating the performance of an open space planner according to an embodiment.

[0011] Figure 7 The illustration shows selected features of each of the four aspects according to one embodiment.

[0012] Figures 8A to 8B These are examples of graphical reports and email reports according to the embodiments.

[0013] Figure 9 This is a block diagram illustrating the process flow of an automatic parameter adjustment framework for a controller used in an autonomous driving vehicle according to one embodiment.

[0014] Figure 10 This is a flowchart illustrating the process of evaluating the performance of an open space planner in an ADV according to an embodiment. Detailed Implementation

[0015] Various embodiments and aspects of this disclosure will be described with reference to the details discussed below, and the accompanying drawings will illustrate the various embodiments. The following description and drawings are illustrative of this disclosure and should not be construed as limiting it. Numerous specific details are described to provide a full understanding of the various embodiments of this disclosure. However, in some cases, well-known or conventional details have not been described in order to provide a concise discussion of embodiments of this disclosure.

[0016] The use of "an embodiment" or "an embodiment" in the specification means that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of this disclosure. The phrase "in one embodiment" appearing in various places throughout the specification does not necessarily refer to the same embodiment.

[0017] According to various embodiments, systems, methods, and media for evaluating open space planners in autonomous vehicles are disclosed. In one embodiment, an exemplary method includes: receiving a configuration file specifying parameters of an autonomous vehicle (ADV), and a log file recorded by the ADV while driving in open space using the open space planner; extracting planning messages and prediction messages from the log file, each extracted message associated with the open space planner. The method further includes: generating features from the planning messages and prediction messages based on specified parameters of the ADV; and calculating statistical metrics based on the features. The statistical metrics are then provided to an automatic tuning framework for adjusting the open space planner.

[0018] In this embodiment, the statistical measure may include the mean, range, and 95th percentile calculated based on multiple characteristics. The statistical measure may be displayed as a visualization on a graphical user interface and / or sent to the user as an email report.

[0019] In an embodiment, extracting planning messages and prediction messages associated with the open space planner further includes: extracting planning messages and prediction messages from a log file; filtering out one or more planning messages that were not generated by the open space planner from the planning messages; filtering out one or more prediction messages that are not related to the open space from the prediction messages; and aligning them based on the timestamps of the remaining planning messages and the remaining prediction messages.

[0020] In one embodiment, features can be extracted directly from the planning message, or features can be calculated based on the extracted features according to the ADV's parameters. These features measure the latency, controllability, safety, and comfort of the trajectory generated by the ADV. The vehicle parameters specified in the configuration file include the ADV's steering ratio, wheelbase, and maximum speed.

[0021] The above embodiments do not exhaustively describe all aspects of the invention. It is contemplated that the invention includes all embodiments that can be practiced from the various embodiments summarized above and all suitable combinations of those embodiments disclosed below.

[0022] autonomous vehicles

[0023] Figure 1 This is a block diagram illustrating an autonomous driving network configuration according to an embodiment of the present disclosure. (Refer to...) Figure 1 Network configuration 100 includes an autonomous vehicle (ADV) 101, which can be communicatively coupled to one or more servers 103 to 104 via network 102. Although only one ADV is shown, multiple ADVs can be coupled to each other and / or coupled to servers 103 to 104 via network 102. Network 102 can be any type of network, such as a local area network (LAN), a wide area network (WAN) (e.g., the Internet), a cellular network, a satellite network, or a combination thereof, wired or wireless. One or more servers 103 to 104 can be any kind of server or server cluster, such as a web server or cloud server, an application server, a backend server, or a combination thereof. Servers 103 to 104 can be data analytics servers, content servers, traffic information servers, map and point of interest (MPOI) servers, or location servers, etc.

[0024] ADV refers to a vehicle that can be configured to operate in an autonomous mode, in which the vehicle navigates its environment with little or no input from the driver. Such an ADV may include a sensor system with one or more sensors configured to detect information about the environment in which the vehicle operates. The vehicle and one or more associated controllers use the detected information to navigate the environment. ADV 101 can operate in manual mode, fully autonomous mode, or partially autonomous mode.

[0025] In one embodiment, ADV 101 includes, but is not limited to, an autonomous driving system (ADS) 110, a vehicle control system 111, a wireless communication system 112, a user interface system 113, and a sensor system 115. ADV 101 may further include certain common components found in ordinary vehicles, such as an engine, wheels, steering wheel, transmission, etc., which can be controlled by the vehicle control system 111 and / or ADS 110 using various communication signals and / or commands (e.g., acceleration signals or commands, deceleration signals or commands, steering signals or commands, braking signals or commands, etc.).

[0026] Components 110 to 115 can be communicatively coupled to each other via interconnects, buses, networks, or combinations thereof. For example, components 110 to 115 can be communicatively coupled to each other via a Controller Area Network (CAN) bus. The CAN bus is a vehicle bus standard designed to allow microcontrollers and devices to communicate with each other in applications without a host computer. It is a message-based protocol originally designed for multiplexing electrical wiring in automobiles, but it is also used in many other environments.

[0027] Now refer to Figure 2 In one embodiment, the sensor system 115 includes, but is not limited to, one or more cameras 211, a Global Positioning System (GPS) unit 212, an Inertial Measurement Unit (IMU) 213, a radar unit 214, and a Light Detection and Ranging (LIDAR) unit 215. The GPS system 212 may include a transceiver capable of operating to provide information about the ADV's location. The IMU unit 213 may sense changes in the ADV's position and orientation based on inertial acceleration. The radar unit 214 may represent a system that uses radio signals to sense objects within the ADV's local environment. In some embodiments, in addition to sensing objects, the radar unit 214 may additionally sense the velocity and / or heading of objects. The LIDAR unit 215 may use lasers to sense objects in the ADV's environment. The LIDAR unit 215 may include one or more laser sources, a laser scanner, and one or more detectors, as well as other system components. The camera 211 may include one or more devices for capturing images of the environment surrounding the ADV. The camera 211 may be a still camera and / or a video camera. The camera may be mechanically movable, for example, by mounting the camera on a rotating and / or tilting platform.

[0028] The sensor system 115 may further include other sensors, such as sonar sensors, infrared sensors, steering sensors, throttle sensors, brake sensors, and audio sensors (e.g., microphones). The audio sensor may be configured to capture sound from the environment surrounding the ADV. The steering sensor may be configured to sense the steering angle of the steering wheel, the vehicle's wheels, or a combination thereof. The throttle and brake sensors sense the vehicle's throttle and brake positions, respectively. In some cases, the throttle and brake sensors may be integrated into an integrated throttle / brake sensor.

[0029] In one embodiment, the vehicle control system 111 includes, but is not limited to, a steering unit 201, a throttle unit 202 (also referred to as an acceleration unit), and a braking unit 203. The steering unit 201 is used to adjust the direction or heading of the vehicle. The throttle unit 202 is used to control the speed of a motor or engine, which in turn controls the speed and acceleration of the vehicle. The braking unit 203 is used to decelerate the vehicle by providing friction to slow down the wheels or tires. Note that, as Figure 2 The components shown can be implemented in hardware, software, or a combination thereof.

[0030] Return to reference Figure 1 The wireless communication system 112 allows communication between the ADV 101 and external systems (e.g., devices, sensors, other vehicles, etc.). For example, the wireless communication system 112 can communicate wirelessly with one or more devices directly or via a communication network, such as communicating wirelessly with servers 103 to 104 via network 102. The wireless communication system 112 can use any cellular communication network or wireless local area network (WLAN), for example, using WiFi to communicate with another component or system. The wireless communication system 112 can communicate directly with devices (e.g., passenger mobile devices, display devices within vehicle 101, speakers, etc.), for example, using infrared links, Bluetooth, etc. The user interface system 113 can be part of peripheral devices implemented within vehicle 101, including, for example, a keyboard, touchscreen display, microphone, and speakers.

[0031] Some or all of the functions of ADV 101 can be controlled or managed by ADS 110, especially when operating in autonomous driving mode. ADS 110 includes the necessary hardware (e.g., one or more processors, memory, storage devices) and software (e.g., operating system, planning and route planning programs) to receive information from sensor system 115, control system 111, wireless communication system 112, and / or user interface system 113, process the received information, plan a route or path from the origin to the destination, and then drive vehicle 101 based on the planning and control information. Alternatively, ADS 110 can be integrated with vehicle control system 111.

[0032] For example, a passenger can specify the start and destination of their trip via a user interface. The ADS 110 obtains trip-related data. For instance, the ADS 110 can obtain location and route data from an MPOI server, which may be part of servers 103 to 104. The location server provides location services, and the MPOI server provides map services and points of interest (POIs) for certain locations. Alternatively, such location and MPOI information can be cached locally in the ADS 110's persistent storage.

[0033] As ADV 101 moves along the route, ADS 110 can also obtain real-time traffic information from a Traffic Information System (TIS) or server. Note that servers 103 to 104 can be operated by a third-party entity. Alternatively, the functionality of servers 103 to 104 can be integrated with ADS 110. Based on real-time traffic information, MPOI information, location information, and real-time local environmental data (e.g., obstacles, objects, nearby vehicles) detected or sensed by sensor system 115, ADS 110 can plan an optimal route and drive vehicle 101 according to the planned route, for example via control system 111, to safely and efficiently reach the designated destination.

[0034] Server 103 may be a data analysis system that performs data analysis services for various clients. In one embodiment, data analysis system 103 includes a data collector 121 and a machine learning engine 122. Data collector 121 collects driving statistics 123 from various vehicles (ADVs or conventional vehicles driven by human drivers). Driving statistics 123 include information indicating issued driving commands (e.g., accelerator commands, braking commands, steering commands) and vehicle responses (e.g., speed, acceleration, deceleration, direction) captured by the vehicle's sensors at different points in time. Driving statistics 123 may further include information describing the driving environment at different points in time, such as route (including starting and destination locations), MPOI, road conditions, weather conditions, etc.

[0035] Based on driving statistics 123, machine learning engine 122 generates or trains a set of rules, algorithms, and / or prediction models 124 for various purposes. Algorithm 124 can then be uploaded to ADV for use in real time during autonomous driving.

[0036] Server 103 may also include an open space planner analysis tool 126, which can evaluate the performance of the open space planner in the ADV and generate statistical performance metrics based on multiple performance characteristics extracted or calculated from the ADV's log files. The statistical performance metrics can be provided to a parameter tuning framework 128, which can use the statistical performance metrics to automatically and iteratively tune the parameters of the open space planner.

[0037] Figure 3A and Figure 3B This is a block diagram illustrating an example of an autonomous driving system used with an ADV according to one embodiment. System 300 can be used as... Figure 1 This is implemented as part of ADV 101, including but not limited to ADS 110, control system 111, and sensor system 115. (See reference...) Figures 3A to 3B The ADS 110 includes, but is not limited to, a positioning module 301, a perception module 302, a prediction module 303, a decision-making module 304, a planning module 305, a control module 306, a route planning module 307, an open space planner 308, and a driving recorder 309.

[0038] Some or all of modules 301 to 309 may be implemented in software, hardware, or a combination thereof. For example, these modules may be installed in persistent storage device 352, loaded into memory 351, and executed by one or more processors (not shown). Note that some or all of these modules may be compatible with... Figure 2 Some or all modules of the vehicle control system 111 are communicatively coupled or integrated. Some of the modules 301 to 309 can be integrated together as an integrated module.

[0039] The positioning module 301 determines the current location of the ADV 300 (e.g., using GPS unit 212) and manages any data related to the user's trip or route. The positioning module 301 (also called the map and route module) manages any data related to the user's trip or route. The user can log in, for example, via a user interface and specify the start and destination of the trip. The positioning module 301 transmits data such as map and route data 311 to other components of the ADV 300 to obtain trip-related data. For example, the positioning module 301 can obtain location and route data from a location server and a map and POI (MPOI) server. The location server provides location services, and the MPOI server provides map services and POIs for certain locations; the location services, map services, and POIs for certain locations can be cached as part of the map and route data 311. As the ADV 300 moves along a route, the positioning module 301 can also obtain real-time traffic information from a traffic information system or server.

[0040] Based on sensor data provided by sensor system 115 and positioning information obtained by positioning module 301, perception module 302 determines the perception of the surrounding environment. The perception information can represent the perception a typical driver would have of the area around the vehicle being driven. Perception may include lane configurations, traffic light signals, the relative position of other vehicles, pedestrians, buildings, crosswalks, or other traffic-related signs in the form of objects (e.g., stop signs, yield signs). Lane configurations include information describing one or more lanes, such as the shape of the lanes (e.g., straight or curved), lane width, number of lanes in the road, one-way or two-way lanes, merging or splitting lanes, exiting lanes, etc.

[0041] The perception module 302 may include a computer vision system or the functionality of a computer vision system to process and analyze images captured by one or more cameras to identify objects and / or features in the ADV's environment. Objects may include traffic signals, road boundaries, other vehicles, pedestrians and / or obstacles, etc. The computer vision system may use object recognition algorithms, video tracking, and other computer vision techniques. In some embodiments, the computer vision system may map the environment, track objects, and estimate the object's velocity, etc. The perception module 302 may also detect objects based on additional sensor data provided by other sensors such as radar and / or LIDAR.

[0042] For each object in the set, prediction module 303 predicts how the object will behave in the given environment. Prediction is performed based on perception data of the driving environment at that point in time, according to a set of map / route information 311 and traffic rules 312. For example, if the object is a vehicle traveling in the opposite direction and the current driving environment includes an intersection, prediction module 303 will predict whether the vehicle is likely to move straight ahead or turn. If the perception data indicates that there are no traffic lights at the intersection, prediction module 303 can predict that the vehicle may have to come to a complete stop before entering the intersection. If the perception data indicates that the vehicle is currently in a left-turn-only lane or a right-turn-only lane, prediction module 303 can predict that the vehicle is more likely to make a left turn or a right turn, respectively.

[0043] For each object in the set, decision module 304 makes a decision about how to deal with that object. For example, for a specific object (e.g., another vehicle on an intersecting road) and its metadata describing that object (e.g., speed, direction, turning angle), decision module 304 determines how to encounter that object (e.g., overtake, yield, stop, pass). Decision module 304 can make such decisions based on a set of rules (e.g., traffic rules or driving rules 312), which can be stored in persistent storage device 352.

[0044] Route planning module 307 is configured to provide one or more routes or paths from a starting point to a destination. For a given trip from a starting location to a destination location, for example, received from a user, route planning module 307 obtains route and map information 311 and determines all possible routes or paths from the starting location to the destination location. Route planning module 307 can generate reference lines in the form of topographic maps for each of the routes it determines from the starting location to the destination location. A reference line is an ideal route or path that is not disturbed by any other factors such as other vehicles, obstacles, or traffic conditions. That is, if there are no other vehicles, pedestrians, or obstacles on the road, the ADV should follow the reference line exactly or closely. The topographic map is then provided to decision module 304 and / or planning module 305. Decision module 304 and / or planning module 305 examine all possible routes to select and modify one of the optimal routes based on other data provided by other modules (e.g., traffic conditions from positioning module 301, driving environment perceived by perception module 302, and traffic conditions predicted by prediction module 303). The actual path or route used to control ADV may be close to or different from the reference line provided by the route planning module 307, depending on the specific driving conditions at that point in time.

[0045] Based on decisions made for each perceived object, the planning module 305 uses reference lines provided by the route planning module 307 as a basis to plan the ADV's path, route, or trajectory, as well as driving parameters (e.g., distance, speed, and / or turning angle). That is, for a given object, the decision module 304 decides how to respond to the object, while the planning module 305 determines how to do so. For example, for a given object, the decision module 304 may decide to pass the object, while the planning module 305 may determine whether to pass to the left or right of the object. The planning module 305 generates planning and control data, including information describing how the vehicle 300 will move in the next movement cycle (e.g., the next route / path segment). For example, the planning and control data may instruct the vehicle 300 to move 10 meters at 30 miles per hour (mph) and then change lanes to the right at 25 mph.

[0046] Based on planning and control data, control module 306 controls and drives the ADV by sending appropriate commands or signals to vehicle control system 111 via CAN bus module 321, according to the trajectory (also known as route or path) defined by the planning and control data. The planning and control data includes sufficient information to drive the vehicle from one point to another along the route or path at different times using appropriate vehicle settings or driving parameters (e.g., throttle command, braking command, steering command).

[0047] In one embodiment, the planning phase is performed over multiple planning cycles (also known as driving cycles), for example, within each 100-millisecond (ms) time interval. For each planning cycle or driving cycle, one or more control commands are issued based on planning and control data. That is, for every 100ms, the planning module 305 plans the next route segment or path segment, including, for example, the target location and the time required for the ADV to reach that target location. Alternatively, the planning module 305 may further specify specific speeds, directions, and / or steering angles, etc. In one embodiment, the planning module 305 plans a route segment or path segment for the next predetermined time period (e.g., 5 seconds). For each planning cycle, the planning module 305 plans a target location for the current cycle (e.g., the next 5 seconds) based on the target location planned in the previous cycle. The control module 306 then generates one or more control commands (e.g., throttle control commands, braking control commands, steering control commands) based on the planning and control data of the current cycle.

[0048] Note that the decision module 304 and the planning module 305 can be integrated into an integrated module. The decision module 304 / planning module 305 may include a navigation system or the functionality of a navigation system to determine the driving path of the ADV. For example, the navigation system may determine a series of speeds and directional headings to influence the movement of the ADV along a path that substantially avoids perceived obstacles, while ensuring that the ADV generally follows a road-based path leading to the final destination. The destination can be set based on user input via the user interface system 113. The navigation system can dynamically update the driving path as the ADV operates. The navigation system can combine data from a GPS system with one or more maps to determine the driving path of the ADV 101. See below. Figure 4 To describe the open space planner 308.

[0049] Driving recorder 309 records driving data from at least three data channels (control channel, chassis channel, and positioning channel) of the ADV control system. The control channel generates information about control commands to control the ADV's systems, such as braking, throttle, and steering. The chassis channel generates information from various sensors (e.g., accelerometers) and readings of the actual position or actuation of the braking, throttle, and steering systems. The positioning channel generates information about the ADV's actual position and heading, referencing standard references such as high-definition (HD) maps or Global Positioning Satellite (GPS) systems. Driving data can be recorded at approximately 100 frames per second (fps) or about 10 milliseconds (ms). Each driving record has a timestamp. The timestamp can be an absolute timestamp in the form of hh:mm:ss:ms (hours, minutes, seconds, milliseconds) relative to a start time (e.g., the start of a driving route). In an embodiment, the timestamp can be a frame number relative to a start time (e.g., the start of a driving route). In an embodiment, each driving record may also have a date stamp in addition to the timestamp. The driving recorder 309 can record driving data in both simulated ADV driving scenarios and real-world ADV driving scenarios.

[0050] The driving recorder 309 can write driving records to a non-volatile storage device, such as the driving log storage device 313. The driving log 313 can be automatically or manually uploaded to a server system (e.g., one or more servers 103 to 104) to generate a set of standardized performance metrics that classify the performance of one or more autonomous driving modules of the ADV.

[0051] Figure 4 This is a block diagram illustrating an example of an open space planning module according to one embodiment. The open space planner 308 can generate trajectories for ADVs (Advanced Vehicles) in an open space where there are no reference lines or lanes to follow. Examples of open spaces include parking lots or roads where vehicles perform parallel parking, U-turns, or three-point turns. (See reference...) Figure 4 In one embodiment, the open space planner 308 includes an environment perception module 401, an objective function determiner module 403, a constraint determiner module 405, a bivariate warm-up module 407, a trajectory generator module 409, and a hybrid A* search module 411. The environment perception module 401 can perceive the ADV's environment. The objective function determiner module 403 can determine the optimization model to be optimized (e.g., open space optimization model 421 as...). Figure 3AThe objective function of model 313 is part of the objective function. The constraint determiner module 405 can determine the constraints on the optimization model. Constraints can include inequality constraints, equality constraints, and bounded constraints. The bivariate warm-up module 407 can apply a quadratic programming (QP) solver to the objective function to solve for one or more variables (e.g., bivariate) subject to certain constraints, where the objective function is a quadratic function. The trajectory generator module 409 can generate trajectories based on the solved variables. The hybrid A* search module 411 can use search algorithms such as the A* search algorithm or a hybrid A* search algorithm to search for initial trajectories (zigzag, non-smooth trajectories that do not consider observed obstacles).

[0052] Open Space Planner Adjustment

[0053] Figure 5 This is a high-level flowchart illustrating the process of adjusting the parameters of an open space planner according to an embodiment.

[0054] like Figure 5 As shown, the log file 501 generated by ADV 101 can be uploaded to the Open Space Planner Analysis Tool 126, which can be a cloud application running server 104. The Open Space Planner Analysis Tool 126 can generate statistical metrics from the log file 501 and provide the statistical metrics 501 to the parameter tuning framework 128, which can use the statistical metrics to automatically adjust the parameters of the Open Space Planner 308 in ADV 101. The log file 501 may contain driving records recorded by the driving recorder 309 and stored in the driving log storage device 313. In one embodiment, the log file may include the output of some of the autonomous driving modules 301 to 309 (e.g., prediction module 303, planning module 305, and control module 306) for each frame during simulation or road testing.

[0055] Figure 6 This is a system for evaluating the performance of an open space planner, according to an embodiment. More specifically, Figure 6 The illustration shows how the statistical performance metrics specific to the open space planner are generated by the open space planner analysis tool 126.

[0056] In one embodiment, the open space planner analysis tool 126 can be integrated with a simulation service (e.g., Baidu). TMThe Apollo Simulation Service and various data services (e.g., calibration services) work together. The Simulation Service can be an open service that simulates autonomous vehicles driving in a virtual environment, generating log files in an Apollo-specific format, which are then passed to the Open Space Planner Analysis Tool 126 for evaluation. Statistical performance metrics calculated by Tool 126 can be stored in the Simulation Service for data visualization and analysis on the website.

[0057] The data service can be a cloud service based on user-specific road test data (e.g., vehicle calibration service). Within this cloud pipeline, the Open Space Planner Analysis Tool 126 can be used to evaluate the large amounts of road test data uploaded by users.

[0058] In one embodiment, the open space planner analysis tool 126 can directly receive recording files from the autonomous vehicle, including those recorded by a driving recorder (e.g., driving recorder 309) and the vehicle specifications that generated the recording files. The open space planner analysis tool 126 may also include multiple standard interfaces, allowing users to upload various types of information as input to the open space planner analysis tool 126.

[0059] For example, the open space planner analysis tool 126 may have an interface that allows users to upload their own road test data, and an interface that allows users to provide specifications of the vehicle that generated the road test data. Examples of vehicle specifications may include the type of autonomous vehicle (e.g., brand, model, revision number), steering ratio, wheelbase, and maximum speed. In embodiments, the record file may be in a specific format, such as Baidu. TM Apollo data format.

[0060] As further shown, the open space planner analysis tool 126 can extract planning messages 601 and prediction messages 603 from the log file 501. As used herein, the messages are the real-time outputs of the corresponding modules when the module is operating an autonomous vehicle.

[0061] For example, planning messages can be generated by the planning module for each frame (e.g., every 100ms) and can include planned paths for subsequent time intervals (e.g., the next 2 seconds). Prediction messages can be generated by the prediction module and can include information such as the vehicle's expected speed at different points on the planned path and the curvature at each point. Control messages can be generated by the control module and can include commands for throttle, braking, and steering.

[0062] Planning message 601 and prediction message 603 can be passed to scene filter 605, which filters out messages irrelevant to the open space planner or open space. Open space planner analysis tool 126 can also align planning and prediction messages based on the timestamps of the planning and prediction messages already passed to scene filter 605. In one embodiment, each planning message can be paired with a corresponding prediction message. Each pair of planning and prediction messages can be passed to feature extraction unit 609 along with user-provided vehicle specification information.

[0063] The feature extraction unit 609 can obtain relevant features, such as point curvature and / or trajectory curvature on the planned path, acceleration, and minimum distance to obstacles. These features can be extracted directly from the planning and prediction messages that have been passed to the scene filter 609, or calculated based on these messages according to vehicle specification information (e.g., vehicle parameters).

[0064] In one embodiment, the extracted or computed features can be categorized into four aspects: latency, controllability, safety, and comfort. Latency can be measured by the time required for the open space planner to generate the planned path, and comfort can be measured by lateral and longitudinal acceleration, as well as jerk. Figure 7 Additional details about this feature are provided in the document.

[0065] Features can be provided to the metric calculation component 611, which can calculate statistical measures based on the features, such as the mean or 95th percentile based on the features.

[0066] A graphical report 613 can be generated for reporting purposes. The graphical report 613 may include various visualization charts. Furthermore, an email report 615 with an analysis summary and one or more text tables can be generated and sent to the user.

[0067] Figure 7 The illustration shows selected features in each of four aspects according to one embodiment. For example... Figure 7As shown, regarding delay 701, features may include selected trajectory delay, zigzag trajectory delay, and phase completion time. Regarding controllability 703, features may include non-shift trajectory length ratio, initial heading difference ratio, normalized curvature ratio, rate of change of curvature ratio, acceleration ratio, deceleration ratio, and longitudinal jerk ratio. Regarding comfort 705, features may include longitudinal jerk ratio, lateral jerk ratio, longitudinal acceleration ratio, lateral acceleration ratio, longitudinal deceleration ratio, lateral deceleration ratio, boundary distance ratio, obstacle distance ratio, and collision time ratio. Regarding safety 707, features may include obstacle distance ratio and collision time ratio. The above features are provided for illustrative purposes. Different features or additional features may be extracted and calculated for each of the above four aspects.

[0068] Figures 8A to 8B These are examples of graphical reports and email reports according to the embodiments.

[0069] Figure 8A This is an example of an email report. As shown in the figure, email reports can be sent to users' email addresses and can include analysis results in tabular form. Figure 8B This is an example of a graphical report on the longitudinal acceleration ratio, showing how the ratio changes over time.

[0070] As mentioned above, statistical performance metrics can be provided to the parameter tuning framework 128, which can use the statistical performance metrics to automatically tune parameters using the configuration of the open space planner.

[0071] Figure 9 This is a block diagram illustrating the process flow of an automatic parameter adjustment framework for an open space planner in ADV according to one embodiment.

[0072] Server 104 may include an ADV driving simulator that can simulate actual driving of an ADV. ADV type parameter 911 may include a list of adjustable parameters for the type of ADV (e.g., compact ADV, truck, or van). The type of ADV can be simulated in the driving simulator.

[0073] The automatic parameter tuning framework 900 includes a tuner component 910, a simulation service 920, and a cost calculation service 930. To achieve high efficiency, the tuner component 910 supports parallel evaluation processes by generating multiple worker threads to simultaneously sample different groups of parameter values ​​for a selected ADV type. The sampling method can be customized based on the parameter optimizer 912 and the sampling strategy. The parameter optimizer 912 can be a Bayesian global optimizer, which can utilize multiple probabilistic models to approximate the objective function, such as Gaussian process regression (GPR) and tree-structured Parzen estimators (TPE).

[0074] Figure 7 The parameters to be sampled by the adjuster component 901 are described in the document. Statistical performance measures (e.g., the mean, 95th percentile, and range of values ​​for each parameter / feature) calculated by the open space planner analysis tool 126 can be used to select relevant parameter values ​​to improve adjustment efficiency.

[0075] For example, if the value of a parameter (e.g., trajectory delay) is within a specific range, the adjuster component 910 will not select parameter values ​​that are not within that specific range when generating multiple parameter groups 915.

[0076] Each of the parameter sets 915 can be combined with a pre-selected set of training scenarios 922 to generate a task, where each task is a unique combination of a parameter set 915 and a training scenario 922.

[0077] The task assignment logic 924 manages tasks and sends requests to the simulation service 920 to execute them. Since the tasks are independent of each other, another round of efficiency improvement is achieved in the simulation service 920 by running all tasks in parallel and returning the execution records to the cost calculation service 930 respectively.

[0078] Upon receiving each execution record, the cost calculation service 930 calculates the task score 920. It also obtains a weighted average score 935 for specific parameter value groups 915 across all training scenarios 922. The average score is fed back to the adjuster component 910 for optimization by the parameter optimizer 912 in the next iteration.

[0079] In an embodiment, for each adjustable parameter in the sampled new parameter set 915, the parameter optimizer 912 selects an initial (“first”) value. The initial value for each adjustable parameter 915 can be randomly selected within a range of values ​​for the adjustable parameters. Performance metrics calculated by the open space planner analysis tool 126 may include such a range of values.

[0080] The parameter optimizer 912 iterates the data stream a predetermined fixed number of times. Each iteration produces a single weighted score 935, which is used by the parameter optimizer 912 as a target to modify the sampled parameters 915 for the next iteration of the optimizer. After a fixed number of iterations have been performed, the parameter optimizer 912 determines the optimal value for each of the plurality of adjustable parameters 915. In subsequent iterations, the parameter optimizer 912 may modify the values ​​of the plurality of adjustable parameters at each iteration of the optimization operation described herein. In an embodiment, the parameter optimizer 912 may use the weighted score 935 to modify the values ​​of the plurality of adjustable parameters for the next iteration of the parameter optimizer 912.

[0081] The parameter optimizer 912 can be configured to optimize a predetermined fixed number of adjustable parameter sets 915 (also referred to as "sampled new parameters 915"), such as sampled new parameter sets 915A to 915C. Optimization can be performed simultaneously, in parallel, and independently on each sampled new parameter set 915A to 915C. Optimization may include... Figure 9 The optimization process is repeated a predetermined fixed number of times. The predetermined fixed number of optimizer iterations for each of the sampled new parameters 915A..915C can be the same number of optimizer iterations, such that when each sampled new parameter set 915A..915C completes its fixed number of optimization iterations, the parameter optimizer 912 can use the weighted score 935A..935C of each sampled new parameter set 915A..915C to select the best sampled new parameter set 915A..915C.

[0082] The cost calculation service 930 may include a database of training scenarios 922. Training scenarios 922 may include thousands of different driving scenarios. In an embodiment, these include various driving scenarios in open spaces, such as making a low-speed left turn and a low-speed right turn in a parking lot.

[0083] Task assignment 924 manages and schedules simulations 925A to 925C for each of the selected driving scenarios 922 within a set of sampled new parameter groups 915A to 915C. For example, if there are 3 sampled new parameter groups and 10 selected driving scenarios, the total number of simulations scheduled can be 40.

[0084] For each of the 40 simulations, simulation service 920 can execute simulation task 925, which may include operations performed concurrently on multiple threads. For each simulation task, cost calculation service 930 can, according to... Figure 7 The performance metrics described herein generate a score 932 that measures the performance of the ADV simulation 925. Therefore, for a sampled new parameter set 915A, the cost calculation service 930 can provide a score calculation for each simulation in simulations 925A1...925A14, where the scores are 932A1...932A14. The cost calculation service 930 can also use the values ​​of the sampled new parameter set 915A for all 40 simulations of the ADV simulation to provide a single weighted score 935A representing the ADV performance.

[0085] In this embodiment, the weights used to generate the weighted score 935 reflect the relative importance of some of the multiple metrics used to generate the score calculation 932. For example, the weight of station endpoint error in a driving scenario may be higher than the weight of average speed error. The weight of safety error may be higher than the weight of passenger comfort error, and the weight of passenger comfort error may be higher than the weight of the frequency of use of control devices (e.g., brakes, steering, or throttle).

[0086] Cost calculation service 930 provides weighted scores 935A to 935C to parameter optimizer 912. The parameter optimizer can use the weighted scores 935A to modify the sampled new parameter 915A for the next iteration (“repetition”) to find the optimal value of the sampled new parameter 915A. Similarly, parameter optimizer 912 can use weighted scores 935B to modify the sampled new parameter 915B for the next iteration to find the optimal value of the sampled new parameter 915B. Additionally, parameter optimizer 912 can use weighted scores 935C to modify the sampled new parameter 915C for the next iteration to find the optimal value of the sampled new parameter 915C.

[0087] In an embodiment, at the end of a configurable predetermined fixed number of iterations of the parameter optimizer 912, the optimal sampled new parameter set 915 can be selected from three (3) sampled new parameter sets 915A...915C, and the optimal sampled new parameter set 915 can be downloaded to a physical real-world ADV with an ADV type for the sampled new parameters 915A...915C to navigate an ADV with that ADV type. In an embodiment, each of the sampled new parameters 915A...915C can be used for a different ADV type. Each sampled new parameter set is optimized upon completion of a fixed number of iterations of the parameter optimizer 912. Each of the optimized sampled new parameter sets 915A...915C can be downloaded to a physical real-world ADV of an ADV type for the sampled new parameter set, and the values ​​of the optimized sampled new parameter sets for the ADV type can be used to navigate each of the physical real-world ADVs.

[0088] For each sampled new parameter set 915, the optimal set of values ​​for the sampled new parameter set can be one or more of the following: (1) the values ​​of the parameters in the sampled new parameter set 915 at the end of a fixed number of iterations of the parameter optimizer 912, (2) the values ​​of the parameter set in the sampled new parameter set 915 at the end of a fixed number of iterations (since this value will be modified by the parameter optimizer 912 if the parameter optimizer 912 has one or more iterations), or (3) the values ​​of the sampled new parameter set 915 after the iteration of the parameter optimizer 912 if the difference between the weighted score 935 of the current iteration and the weighted score 935 of the previous iteration of the parameter optimizer 912 is less than a predetermined threshold amount. In an embodiment where all sampled new parameter groups 915A..915C are associated with the same type of ADV, the optimal value in the sampled new parameter groups 915A..915C can be the sampled new parameter group 915 with the best weighted score 935 after optimization of each sampled new parameter group in the sampled new parameter groups 915A..915C is completed.

[0089] Figure 10 This is a flowchart illustrating a process 1000 for evaluating the performance of an open space planner in an ADV according to an embodiment. Process 1000 can be executed by processing logic, which may include software, hardware, or a combination thereof. For example, process 1000 can be performed by… Figure 1 and Figure 6 The open space planner analysis tool 126 described herein is used to perform this.

[0090] like Figure 10 As shown, in operation 1001, the processing logic receives a log file recorded by the ADV while driving in open space using the open space planner, and a configuration file specifying the parameters of the ADV. In operation 1002, the processing logic extracts multiple planning messages and multiple prediction messages from the log file, wherein each extracted message is associated with the open space planner. In operation 1003, the processing logic generates multiple features from the planning messages and prediction messages based on the specified parameters of the ADV. In operation 1004, the processing logic calculates multiple statistical measures based on the multiple features. In operation 1005, the processing logic provides the multiple statistical measures to the automatic adjustment framework for adjusting the open space planner.

[0091] Note that some or all of the components shown and described above may be implemented in software, hardware, or a combination thereof. For example, such a component may be implemented as software installed and stored in a persistent storage device, which may be loaded into memory and executed by a processor (not shown) to perform the processes or operations described throughout this application. Alternatively, such a component may be implemented as executable code programmed or embedded in dedicated hardware such as an integrated circuit (e.g., an application-specific integrated circuit or ASIC), a digital signal processor (DSP), or a field-programmable gate array (FPGA), accessible from an application via a corresponding driver and / or operating system. Furthermore, such a component may be implemented as dedicated hardware logic within a processor or processor core as part of an instruction set accessible by software components via one or more specific instructions.

[0092] The preceding sections have already described some aspects of the algorithms and symbolic representations for manipulating data bits within computer memory. These algorithmic descriptions and representations are the most efficient way for those skilled in the art of data processing to communicate the essence of their work to others skilled in the art. Here, an algorithm is generally considered a self-consistent sequence of operations that produces a desired result. An operation is one that requires physical manipulation of a physical quantity.

[0093] However, it should be remembered that all of these and similar terms are associated with appropriate physical quantities and are merely convenient labels applied to those quantities. Unless otherwise expressly stated from the foregoing discussion, it should be understood that throughout the specification, the discussion using terms such as those set forth in the appended claims refers to the actions and processes of a computer system or similar electronic computing device that manipulate data represented as physical (electronic) quantities within registers and memories of a computer system and convert that data into other data represented similarly as physical quantities within computer system memories or registers or other such information storage devices, transmission devices, or display devices.

[0094] Embodiments of this disclosure also relate to an apparatus for performing the operations described herein. Such a computer program is stored in a non-transitory computer-readable medium. A machine-readable medium includes any means for storing information in a form that can be read by a machine (e.g., a computer). For example, machine-readable (e.g., computer-readable) media include machine-readable (e.g., computer-readable) storage media (e.g., read-only memory (“ROM”), random access memory (“RAM”), disk storage media, optical storage media, flash memory devices).

[0095] The processes or methods depicted in the foregoing figures can be executed by processing logic including hardware (e.g., circuits, special-purpose logic, etc.), software (e.g., embodied on a non-transitory computer-readable medium), or a combination of both. While the processes or methods have been described above in a certain order, it should be understood that some of the operations described can be performed in a different order. Furthermore, some operations can be performed in parallel rather than sequentially.

[0096] The embodiments of this disclosure are described without reference to any particular programming language. It will be understood that the teachings of the embodiments of this disclosure described herein can be implemented using various programming languages.

[0097] In the foregoing description, embodiments of the present disclosure have been described with reference to specific exemplary embodiments. It will be apparent that various modifications may be made to the present disclosure without departing from the broader spirit and scope of the disclosure as set forth in the appended claims. Accordingly, the description and drawings should be regarded as illustrative rather than restrictive.

Claims

1. A computer-implemented method for evaluating an open space planner for an autonomous vehicle (ADV), the method comprising: The analysis application executed by the processor receives a log file and a configuration file specifying the parameters of the ADV, the log file having previous driving records recorded by the ADV while driving in open space using the open space planner; The analysis application extracts multiple planning messages and multiple prediction messages from the log file, wherein each extracted message is associated with the open space planner; The analysis application generates multiple features from the multiple planning messages and the multiple prediction messages according to specified parameters of the ADV, wherein the multiple features include: The characteristics that measure the delay required for the open space planner to generate a planned path include at least one of the following: selected trajectory delay, zigzag trajectory delay, or stage completion time; The characteristics that measure the controllability of the planned path include at least one of the following: non-shift trajectory length ratio, initial heading difference ratio, normalized curvature ratio, rate of change of curvature ratio, acceleration ratio, deceleration ratio, or longitudinal jerk. The characteristics used to measure the safety of the planned path include at least one of obstacle distance ratio and collision time ratio; and The features for measuring the comfort of the planned path include at least one of the following: longitudinal jerk ratio, lateral jerk ratio, longitudinal acceleration ratio and lateral acceleration ratio, longitudinal deceleration ratio, lateral deceleration ratio, boundary distance ratio, obstacle distance ratio, and collision time ratio. The analysis application calculates multiple statistical measures from the multiple features, wherein the statistical measures include a range of values ​​calculated from the multiple features; and The analysis application provides the multiple statistical measures to an auto-tuning framework to adjust the open space planner, which includes multiple adjustable parameters. The auto-tuning framework adjusts the open space planner in the following manner: The numerical range of the multiple adjustable parameters is determined based on the multiple statistical measures, so as to determine the initial value of the multiple adjustable parameters; The simulation task is executed based on the determined initial values ​​of the plurality of adjustable parameters; Based on the feature dimensions of the multiple features, the score of the simulation task is determined; and The parameter optimizer is used to adjust the multiple adjustable parameters based on the multiple statistical metrics and the scores of the simulation task.

2. The method of claim 1, wherein, Extracting the plurality of planning messages and the plurality of prediction messages associated with the open space planner further includes: Extract all planning and forecasting messages from the record file; Filter out one or more planning messages that were not generated by the open space planner from all the planning messages; Filter out one or more prediction messages that are irrelevant to the open space planner from all the prediction messages; and The remaining planning messages and the remaining prediction messages are aligned based on the timestamps of the remaining planning messages and the remaining prediction messages in all the planning messages.

3. The method of claim 1, wherein, Generating the plurality of features includes extracting one or more features from the plurality of planning messages, and calculating one or more features based on one or more extracted features and the parameters of the ADV.

4. The method of claim 1, wherein, The parameters of the ADV specified in the configuration file include the ADV's steering ratio, wheelbase, and maximum speed.

5. The method of claim 1, wherein, The statistical measures include the mean and 95th percentile calculated from the plurality of features.

6. The method of claim 1, wherein, The statistical metrics are displayed as visual charts on the graphical user interface and sent to users as email reports.

7. A non-transitory machine-readable medium having instructions stored thereon, the instructions, when executed by a processor, causing the processor to perform operations of evaluating an open-space planner for an autonomous vehicle (ADV), the operations including: Receive a log file and a configuration file specifying the parameters of the ADV, the log file having previous driving records recorded by the ADV while driving in open space using the open space planner; Multiple planning messages and multiple prediction messages are extracted from the record file, wherein each extracted message is associated with the open space planner; Multiple features are generated from the multiple planning messages and the multiple prediction messages according to the specified parameters of the ADV, wherein the multiple features include: The characteristics that measure the delay required for the open space planner to generate a planned path include at least one of the following: selected trajectory delay, zigzag trajectory delay, or stage completion time; The characteristics that measure the controllability of the planned path include at least one of the following: non-shift trajectory length ratio, initial heading difference ratio, normalized curvature ratio, rate of change of curvature ratio, acceleration ratio, deceleration ratio, or longitudinal jerk. The characteristics used to measure the safety of the planned path include at least one of obstacle distance ratio and collision time ratio; and The features for measuring the comfort of the planned path include at least one of the following: longitudinal jerk ratio, lateral jerk ratio, longitudinal acceleration ratio and lateral acceleration ratio, longitudinal deceleration ratio, lateral deceleration ratio, boundary distance ratio, obstacle distance ratio, and collision time ratio; Multiple statistical measures are calculated from the plurality of features, wherein the statistical measures include a range of values ​​calculated from the plurality of features; and The multiple statistical measures are provided to the automatic adjustment framework to adjust the open space planner, which includes multiple adjustable parameters. The automatic adjustment framework adjusts the open space planner in the following manner: The numerical range of the multiple adjustable parameters is determined based on the multiple statistical measures, so as to determine the initial value of the multiple adjustable parameters; The simulation task is executed based on the determined initial values ​​of the plurality of adjustable parameters; Based on the feature dimensions of the multiple features, the score of the simulation task is determined; and The parameter optimizer is used to adjust the multiple adjustable parameters based on the multiple statistical metrics and the scores of the simulation task.

8. The non-transitory machine readable medium of claim 7, wherein, Extracting the plurality of planning messages and the plurality of prediction messages further includes: Extract all planning and forecasting messages from the record file; Filter out one or more planning messages that were not generated by the open space planner from all the planning messages; Filter out one or more prediction messages that are irrelevant to the open space from all the prediction messages; and The remaining planning messages and the remaining prediction messages are aligned based on the timestamps of the remaining planning messages and the remaining prediction messages in all the planning messages.

9. The non-transitory machine readable medium of claim 7, wherein, Generating the plurality of features includes extracting one or more features from the plurality of planning messages, and calculating one or more features based on one or more extracted features and the parameters of the ADV.

10. The non-transitory machine readable medium of claim 7, wherein, The parameters of the ADV specified in the configuration file include the ADV's steering ratio, wheelbase, and maximum speed.

11. The non-transitory machine readable medium of claim 7, wherein, The statistical measures include the mean and 95th percentile calculated from the plurality of features.

12. The non-transitory machine readable medium of claim 7, wherein, The statistical metrics are displayed as visual charts on the graphical user interface and sent to users as email reports.

13. A data processing system, comprising: processor; as well as A memory coupled to the processor, the memory storing instructions that, when executed by the processor, cause the processor to perform operations of evaluating an open space planner for an autonomous vehicle (ADV), the operations including: Receive a log file and a configuration file specifying the parameters of the ADV, the log file having previous driving records recorded by the ADV while driving in open space using the open space planner, wherein the open space indication does not include the space of reference lines or lane lines that the ADV needs to follow; Multiple planning messages and multiple prediction messages are extracted from the record file, wherein each extracted message is associated with the open space planner; Multiple features are generated from the multiple planning messages and the multiple prediction messages according to the specified parameters of the ADV, wherein the multiple features include: The characteristics that measure the delay required for the open space planner to generate a planned path include at least one of the following: selected trajectory delay, zigzag trajectory delay, or stage completion time; The characteristics that measure the controllability of the planned path include at least one of the following: non-shift trajectory length ratio, initial heading difference ratio, normalized curvature ratio, rate of change of curvature ratio, acceleration ratio, deceleration ratio, or longitudinal jerk. The characteristics used to measure the safety of the planned path include at least one of obstacle distance ratio and collision time ratio; and The features for measuring the comfort of the planned path include at least one of the following: longitudinal jerk ratio, lateral jerk ratio, longitudinal acceleration ratio and lateral acceleration ratio, longitudinal deceleration ratio, lateral deceleration ratio, boundary distance ratio, obstacle distance ratio, and collision time ratio. Multiple statistical measures are calculated from the plurality of features, wherein the statistical measures include a range of values ​​calculated from the plurality of features; and Multiple statistical measures are provided to the auto-tuning framework to adjust the open space planner, which includes multiple adjustable parameters. The auto-tuning framework adjusts the open space planner in the following manner: The numerical range of the multiple adjustable parameters is determined based on the multiple statistical measures, so as to determine the initial value of the multiple adjustable parameters; The simulation task is executed based on the determined initial values ​​of the plurality of adjustable parameters; Based on the feature dimensions of the multiple features, the score of the simulation task is determined; and The parameter optimizer is used to adjust the multiple adjustable parameters based on the multiple statistical metrics and the scores of the simulation task.

14. The data processing system of claim 13, wherein, The instructions are configured to run on the ADV or on a cloud server.

15. The data processing system of claim 13, wherein, Extracting the plurality of planning messages and the plurality of prediction messages further includes: Extract all planning and forecasting messages from the record file; Filter out one or more planning messages from all the planning messages that were not generated by the open space planner; Filter out one or more prediction messages that are irrelevant to the open space from all the prediction messages; and The remaining planning messages and the remaining prediction messages are aligned based on the timestamps of the remaining planning messages and the remaining prediction messages in all the planning messages.

16. The data processing system of claim 13, wherein, Generating the plurality of features includes extracting one or more features from the plurality of planning messages, and calculating one or more features based on one or more extracted features and the parameters of the ADV.

17. The data processing system of claim 13, wherein, The parameters of the ADV specified in the configuration file include the ADV's steering ratio, wheelbase, and maximum speed.

18. A computer program product comprising a computer program, wherein, When the computer program is executed by a processor, it implements the method of any one of claims 1-6.

Images