Select test scenarios for evaluating the performance of an automated vehicle

Abstract

The present invention relates to selecting test scenarios for evaluating the performance of an automated vehicle. In an example method, a computer system receives first data representing a plurality of scenarios for estimating the performance of a vehicle during autonomous operation. Additionally, for each scenario, the computer system determines: (i) a first metric that indicates the observed performance of the vehicle in the scenario, where the first metric is determined based on at least one rule; and (ii) a second metric that indicates the degree of information gain associated with the scenario. The computer system selects a subset of the scenarios based on the first metric and the second metric, and outputs second data indicating the subset of the scenarios.

Description

Technical Field

[0001] This specification relates to a computer system for selecting test scenarios for evaluating the performance of an autonomous vehicle. Background Art

[0002] Autonomous vehicles (AVs) are typically tested in multiple scenarios to evaluate the behavior of the AV under various conditions. Based on the results of the tests, the AV can be modified and / or reconfigured to improve its behavior.

[0003] As an example, during development, the autonomous navigation capabilities of an AV can be tested in multiple scenarios, each scenario representing a different combination of roads, obstacles, other vehicles, pedestrians, and traffic flows that the AV must traverse. Based on the results of the tests, the developers can modify and / or reconfigure the AV such that its autonomous navigation capabilities are improved. Summary of the Invention

[0004] According to a first aspect of the present invention, a method includes: receiving, by a computer system, first data representing a plurality of scenarios for estimating the performance of a vehicle during autonomous operation; for each scenario, determining, by the computer system: a first metric that indicates the observed performance of the vehicle in the scenario, wherein the first metric is determined based on at least one rule, and a second metric that indicates the degree of information gain associated with the scenario; selecting, by the computer system, a subset of scenarios based on the first metric and the second metric; and outputting, by the computer system, second data indicating the subset of scenarios.

[0005] According to a second aspect of the present invention, a system includes: at least one processor; and at least one non-transitory computer-readable medium storing instructions that, when executed by the at least one processor, cause the at least one processor to perform operations including: receiving first data representing a plurality of scenarios for estimating the performance of a vehicle during autonomous operation; for each scenario, determining: a first metric that indicates the observed performance of the vehicle in the scenario, wherein the first metric is determined based on at least one rule, and a second metric that indicates the degree of information gain associated with the scenario; selecting a subset of scenarios based on the first metric and the second metric; and outputting second data indicating the subset of scenarios.

[0006] According to a third aspect of the present invention, at least one non-transitory computer-readable medium stores instructions that, when executed by at least one processor, cause the at least one processor to perform operations including: receiving first data representing a plurality of scenarios for estimating the performance of a vehicle during autonomous operation; for each scenario, determining: a first metric that indicates the observed performance of the vehicle in that scenario, where the first metric is determined based on at least one rule, and a second metric that indicates the degree of information gain associated with that scenario; selecting a subset of scenarios based on the first metric and the second metric; and outputting second data indicating the subset of scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 Shows an example of an AV with autonomous capabilities.

[0008] Figure 2 Shows an example "cloud" computing environment.

[0009] Figure 3 Shows a computer system.

[0010] Figure 4 Shows an example architecture of an AV.

[0011] Figure 5 Shows examples of inputs and outputs that a perception module can use.

[0012] Figure 6 Shows an example of a LiDAR system.

[0013] Figure 7 Shows a LiDAR system in operation.

[0014] Figure 8 Shows additional details of the operation of a LiDAR system.

[0015] Figure 9 Shows a block diagram of the relationship between the inputs and outputs of a planning module.

[0016] Figure 10 Shows a directed graph used in path planning.

[0017] Figure 11 Shows a block diagram of the inputs and outputs of a control module.

[0018] Figure 12 Shows a block diagram of the inputs, outputs, and components of a controller.

[0019] Figure 13 Shows an example system for iteratively prototyping an AV and validating the performance of the prototyped AV.

[0020] Figure 14A and Figure 14B illustrates a simplified example of a scenario for testing the performance of an AV.

[0021] Figure 15 illustrates an example process for selecting a set of scenarios for testing the performance of an AV.

[0022] Figure 16 illustrates a simplified relationship between the number of scenarios tested and the total amount of information obtained regarding the performance of the AV.

[0023] Figure 17 illustrates a representation of a Bayesian hierarchical model.

[0024] Figure 18 illustrates an example probability distribution of a random variable X.

[0025] Figure 19 illustrates maps of two example towns for evaluating the performance of an AV.

[0026] Figure 20 illustrates the probability distribution of collisions in all scenarios of an example computerized simulation.

[0027] Figure 21 illustrates the distribution of collisions along a route for six example towns according to an example computerized simulation.

[0028] Figure 22A illustrates an example posterior distribution of unobservable parameters of a fitted Bayesian hierarchical model.

[0029] Figure 22B illustrates an example predictive posterior fit of a Bayesian hierarchical model.

[0030] Figure 23 illustrates a comparison of an example scenario selection algorithm, a Latin hypercube sampling implementation, and several randomly selected information gains described herein.

[0031] Figure 24 illustrates a flowchart of an example process for monitoring and controlling the operation of one or more AVs. Detailed Description

[0032] In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present invention. It will be apparent, however, that the present invention may be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form in order to avoid unnecessarily obscuring the present invention.

[0033] In the drawings, for ease of description, a specific arrangement or order of illustrative elements (such as those representing devices, modules, instruction blocks, and data elements) is shown. However, those skilled in the art should understand that the specific order or arrangement of the illustrative elements in the drawings is not intended to imply a required processing order or sequence, or a separation of processing procedures. In addition, the inclusion of illustrative elements in the drawings is not intended to imply that such elements are required in all embodiments, nor is it intended to imply that the features represented by such elements cannot be included in some embodiments or cannot be combined with other elements in some embodiments.

[0034] In addition, in the drawings, connecting elements, such as solid lines, dashed lines, or arrows, are used to illustrate the connection, relationship, or association between two or more other illustrative elements. The absence of any such connecting element is not intended to imply that a connection, relationship, or association cannot exist. In other words, the connection, relationship, or association between some elements is not shown in the drawings so as not to obscure the present disclosure. In addition, for ease of illustration, a single connecting element is used to represent multiple connections, relationships, or associations between elements. For example, if a connecting element represents the communication of signals, data, or instructions, those skilled in the art should understand that such an element represents one or more signal paths (e.g., a bus) that may be required to affect the communication.

[0035] Reference will now be made in detail to the embodiments, which are illustrated in the accompanying drawings. In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the various embodiments described. However, it will be apparent to those of ordinary skill in the art that the various embodiments described may be practiced without these specific details. In other instances, well-known methods, procedures, components, circuits, and networks have not been described in detail so as not to unnecessarily obscure aspects of the embodiments.

[0036] Several of the features described below may be used independently of each other or in any combination with other features. However, any individual feature may not solve any of the problems discussed above, or may only solve one of the problems discussed above. Some of the problems discussed above may not be fully solved by any one feature described herein. Although headings are provided, information related to specific headings may also be found elsewhere in this specification but not in the section with that heading. The embodiments are described herein according to the following outline:

[0037] 1. General Overview

[0038] 2. System Overview

[0039] 3. AV Architecture

[0040] 4. AV Input

[0041] 5. AV Planning

[0042] 6. AV Control

[0043] 7. AV Performance Evaluation System

[0044] 8. Example Techniques for Selecting Scenarios Based on Information Gain

[0045] General Overview

[0046] Autonomous vehicles (AVs) are typically tested in multiple scenarios to evaluate the behavior of the AV under various conditions. For example, each scenario can specify a particular combination of roads, obstacles, other vehicles, pedestrians, traffic flow, and environmental conditions that the AV must traverse. If the AV exhibits an undesirable behavior in one or more of these scenarios (e.g., if the AV is involved in a collision, violates the rules of the road, etc.), then the behavior of the AV can be modified so that the AV is less likely to exhibit that behavior in the future.

[0047] In some cases, given a particular resource budget, a set of scenarios can be selected to maximize the amount of information obtained about the performance of the AV. For example, the performance of the AV can be represented as a random variable with a particular probability distribution. Additionally, one or more scenarios can be used for testing such that the entropy of this random variable is decreased. Scenarios that decrease the entropy of the random variable by a particular degree when used for testing can be preferentially selected over scenarios that decrease the entropy of the random variable by a relatively small degree when used for testing. Additionally, the selected set of scenarios can be included in a set of standard tests to evaluate the performance of the AV during the development and validation of the AV's autonomous navigation system.

[0048] Some advantages of these techniques include improving the efficiency and rate at which the performance of the AV can be tested.

[0049] For example, in traditional methods, the AV may be tested in a large number of scenarios in a brute-force manner without considering the amount of information obtained from each test scenario. Since some scenarios may provide little probative value in evaluating the performance of the AV and / or considering that other scenarios may be redundant, resources may be wasted when testing these scenarios. Additionally, since each successive scenario tested may provide diminishing returns in evaluating the performance of the AV, a large amount of resources may be invested to obtain relatively less useful information about the performance of the AV.

[0050] In contrast, the techniques described herein enable the performance of the AV to be tested more quickly and efficiently. For example, a limited set of scenarios can be selected for testing such that each scenario in the set provides useful information for evaluating the performance of the AV. Additionally, the set can be selected to mitigate the effects of diminishing returns. Thus, a smaller number of test scenarios (e.g., compared to the number of test scenarios that might be employed in a brute-force approach) can be used to determine the performance of the AV.

[0051] In some implementations, these techniques (e.g., by enabling developers to more quickly identify undesired behaviors of the AV such that the behaviors can be corrected) can accelerate the development and validation of the AV's autonomous navigation system. Additionally, these techniques can be used to determine whether the AV meets certain safety goals or requirements (e.g., a desired safety level).

[0052] System Overview

[0053] Figure 1 An example of an AV 100 with autonomous capabilities is shown.

[0054] As used herein, the term "autonomous capabilities" refers to a function, feature, or facility that enables a vehicle to operate partially or fully without real-time human intervention, including but not limited to fully autonomous AVs, highly autonomous AVs, and conditional AVs.

[0055] As used herein, an AV (autonomous vehicle) is a vehicle with autonomous capabilities.

[0056] As used herein, "vehicle" includes a means of transporting goods or people. For example, cars, buses, trains, airplanes, drones, trucks, ships, vessels, submersibles, airships, etc. A driverless car is an example of a vehicle.

[0057] As used herein, "trajectory" refers to the path or route that navigates an AV from a first spatio-temporal location to a second spatio-temporal location. In an embodiment, the first spatio-temporal location is referred to as the initial location or starting location, and the second spatio-temporal location is referred to as the destination, final location, target, target position, or target location. In some examples, the trajectory consists of one or more segments (e.g., several segments of a road), and each segment consists of one or more blocks (e.g., a portion of a lane or intersection). In an embodiment, the spatio-temporal location corresponds to a real-world location. For example, the spatio-temporal location is a pick-up or drop-off location to pick up or drop off people or goods.

[0058] As used herein, "(one or more) sensors" includes one or more hardware components for detecting information related to the environment surrounding the sensor. Some hardware components may include sensing components (e.g., image sensors, biometric sensors), transmission and / or reception components (e.g., laser or radio wave transmitters and receivers), electronic components (such as analog-to-digital converters), data storage devices (such as RAM and / or non-volatile memory), software or firmware components, and data processing components (such as application-specific integrated circuits), microprocessors, and / or microcontrollers.

[0059] As used herein, "scene description" is a data structure (e.g., a list) or data stream that includes one or more classified or labeled objects detected by one or more sensors on an AV vehicle, or one or more classified or labeled objects provided by a source external to the AV.

[0060] As used herein, "road" is a physical area that can be traversed by a vehicle and can correspond to a named passageway (e.g., a city street, an interstate highway, etc.) or can correspond to an unnamed passageway (e.g., a driveway within a house or office building, a section of a parking lot, a section of an empty parking lot, a dirt road in a rural area, etc.). Because some vehicles (e.g., four-wheel drive pickup trucks, sport utility vehicles (SUVs), etc.) are capable of traversing various physical areas that are not particularly suitable for vehicle travel, a "road" can be any physical area that has not been formally defined as a passageway by any municipality or other government or administrative agency.

[0061] As used herein, "lane" is the portion of a road that can be traversed by a vehicle. Lanes are sometimes identified based on lane markings. For example, a lane can correspond to most or all of the space between lane markings, or can correspond to only a portion of the space between lane markings (e.g., less than 50%). For example, a road with widely spaced lane markings may accommodate two or more vehicles such that one vehicle can pass another without crossing the lane markings, and thus can be interpreted as having lanes that are narrower than the space between the lane markings, or having two lanes between the lane markings. Lanes can also be interpreted in the absence of lane markings. For example, lanes can be defined based on the physical characteristics of the environment (e.g., rocks in a rural area and trees along a boulevard, or natural obstacles to be avoided in an underdeveloped area, for example). Lanes can also be interpreted independently of lane markings or physical characteristics. For example, lanes can be interpreted based on any obstacle-free path in a region that otherwise lacks features that would be interpreted as lane boundaries. In an example scenario, an AV can interpret a lane through an obstacle-free portion of a field or open space. In another example scenario, an AV can interpret lanes through a wide (e.g., wide enough for two or more lanes) road that does not have lane markings. In this scenario, the AV can communicate information related to the lanes to other AVs such that the other AVs can use the same lane information to coordinate path planning among the AVs.

[0062] As used herein, "homotopy" refers to a subset of a set of constraints on the trajectory of an AV that the AV can abide by when traversing a particular route.

[0063] As used herein, "feasible" refers to whether an AV can abide by the constraints in a homotopy when traveling towards a destination.

[0064] "One or more" includes functions performed by one element, functions performed by multiple elements, such as in a distributed manner, several functions performed by one element, several functions performed by several elements, or any combination of the above.

[0065] It will also be understood that although in some instances the terms "first", "second", etc. are used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, without departing from the scope of the various described embodiments, a first contact may be referred to as a second contact, and similarly, a second contact may be referred to as a first contact. Both the first contact and the second contact are contacts, but they are not the same contact.

[0066] The terms used in the specification of the various embodiments described herein are for the purpose of describing particular embodiments only and are not intended to be limiting. As used in the specification of the various embodiments and the appended claims, the singular forms "a", "an", and "the" are also intended to include the plural forms unless the context clearly dictates otherwise. It will also be understood that as used herein, "and / or" refers to and includes any and all possible combinations of one or more of the associated listed items. It will also be understood that when the terms "comprises", "comprising", "includes", and / or "including" are used in this specification, there is specified the presence of the stated features, integers, steps, operations, elements, and / or components, but there is not precluded the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0067] As used herein, depending on the context, the term "if" is optionally understood to mean "when" or "at that time" or "in response to determining" or "in response to detecting". Similarly, depending on the context, the phrase "if it has been determined" or "if [the stated condition or event] has been detected" is optionally understood to mean "upon determining" or "in response to determining" or "at the time of detecting [the stated condition or event]" or "in response to detecting [the stated condition or event]".

[0068] As used herein, an AV system refers to an array of AV and the hardware, software, stored data, and real-time generated data that support AV operations. In an embodiment, the AV system is incorporated within the AV. In an embodiment, the AV system is distributed across several locations. For example, some of the software of the AV system is implemented on a cloud computing environment similar to the cloud computing environment 200 described below with respect to Figure 2 described cloud computing environment 200.

[0069] In general, this document describes technologies applicable to any vehicle having one or more autonomous capabilities, including full AVs, highly automated AVs, and conditionally automated AVs, such as so-called Level 5, Level 4, and Level 3 vehicles, respectively (see SAE International Standard J3016: Taxonomy and Definitions for Terms Related to On-Road Motor Vehicle Automated Driving Systems, which is incorporated herein by reference in its entirety for more details on vehicle autonomy levels). The technologies described in this document are also applicable to partially automated AVs and driver assistance vehicles, such as so-called Level 2 and Level 1 vehicles (see SAE International Standard J3016: Taxonomy and Definitions for Terms Related to On-Road Motor Vehicle Automated Driving Systems). In embodiments, one or more Level 1, Level 2, Level 3, Level 4, and Level 5 vehicle systems can automatically perform certain vehicle operations (e.g., steering, braking, and using maps) under certain operating conditions based on the processing of sensor inputs. The technologies described in this document can benefit vehicles of any level within the range from full AVs to human-operated vehicles.

[0070] AVs have advantages compared to vehicles that require a human driver. One advantage is safety. For example, in 2016, the United States experienced 6 million motor vehicle accidents, 2.4 million people injured, 40,000 people killed, and 13 million vehicle crashes, with an estimated societal cost of over $910 billion. From 1965 to 2015, the number of traffic fatalities per 100 million vehicle miles traveled in the United States has decreased from approximately 6 to approximately 1, in part due to additional safety measures deployed in vehicles. For example, an extra half-second warning related to an impending collision is believed to mitigate 60% of front-to-back collisions. However, passive safety features (e.g., seat belts, airbags) may have reached their limits in improving this number. Thus, active safety measures such as the automatic control of vehicles are a likely next step in improving these statistics. Since human drivers are considered the cause of serious pre-crash events in 95% of collisions, an automated driving system has the potential to achieve better safety outcomes, for example, by: reliably identifying and avoiding emergencies better than humans; making better decisions than humans, complying with traffic laws better than humans, and predicting future events better than humans; and reliably controlling the vehicle better than humans.

[0071] Reference Figure 1 , the AV system 120 causes the AV 100 to operate along a trajectory 198, through an environment 190 to a destination 199 (sometimes referred to as a final location), while avoiding objects (e.g., natural obstacles 191, vehicles 193, pedestrians 192, cyclists, and other obstacles) and complying with road rules (e.g., operating rules or driving preferences).

[0072] In an embodiment, the AV system 120 includes means 101 for receiving and operating on operation commands from a computer processor 146. The term "operation commands" is used to denote executable instructions (or sets of instructions) that cause the vehicle to perform actions (e.g., driving maneuvers). Operation commands can include, without limitation, instructions to cause the vehicle to start moving forward, stop moving forward, start moving backward, stop moving backward, accelerate, decelerate, turn left, and turn right. In an embodiment, the computer processor 146 is similar to the processor 304 described below with reference to Figure 3 Examples of means 101 include a steering controller 102, brakes 103, gear positions, an accelerator pedal or other acceleration control mechanism, windshield wipers, side door locks, window controllers, and turn indicators.

[0073] In an embodiment, the AV system 120 includes sensors 121 for measuring or inferring attributes of the state or condition of the AV 100, such as the position, linear velocity and angular velocity, and linear acceleration and angular acceleration of the AV, as well as the heading (e.g., the direction of the front end of the AV 100). Examples of sensors 121 are GPS, an inertial measurement unit (IMU) that measures both the linear acceleration and angular rate of the vehicle, wheel speed sensors for measuring or estimating wheel slip rate, wheel brake pressure or brake torque sensors, engine torque or wheel torque sensors, and steering angle and angular rate sensors.

[0074] In an embodiment, sensors 121 also include sensors for sensing or measuring attributes of the environment of the AV. For example, monocular or stereo cameras 122 in the visible, infrared, or thermal (or both) spectra, LiDAR 123, RADAR, ultrasonic sensors, time-of-flight (TOF) depth sensors, rate sensors, temperature sensors, humidity sensors, and precipitation sensors.

[0075] In an embodiment, the AV system 120 includes a data storage unit 142 and a memory 144 for storing machine instructions associated with the computer processor 146 or data collected by the sensors 121. In an embodiment, the data storage unit 142 is similar to the ROM 308 or storage device 310 described below with reference to Figure 3 In an embodiment, the memory 144 is similar to the main memory 306 described below. In an embodiment, the data storage unit 142 and the memory 144 store historical, real-time, and / or predictive information about the environment 190. In an embodiment, the stored information includes maps, driving performance, traffic congestion updates, or weather conditions. In an embodiment, data related to the environment 190 is transmitted from a remote database 134 to the AV 100 via a communication channel.

[0076] In an embodiment, the AV system 120 includes communication devices 140 for transmitting attributes measured or inferred about the state and conditions of other vehicles (such as position, linear and angular velocity, linear and angular acceleration, and linear and angular heading) to the AV 100. These devices include vehicle-to-vehicle (V2V) and vehicle-to-infrastructure (V2I) communication devices and devices for wireless communication via a point-to-point or ad hoc network or both. In an embodiment, the communication devices 140 communicate across the electromagnetic spectrum (including radio and optical communication) or other media (such as air and acoustic media). The combination of vehicle-to-vehicle (V2V), vehicle-to-infrastructure (V2I) communication (and in some embodiments one or more other types of communication) is sometimes referred to as vehicle-to-everything (V2X) communication. V2X communication generally complies with one or more communication standards for communication with autonomous vehicles and between AVs.

[0077] In an embodiment, the communication device 140 includes a communication interface. For example, a wired, wireless, WiMAX, Wi-Fi, Bluetooth, satellite, cellular, optical, near-field, infrared, or radio interface. The communication interface transfers data from the remote database 134 to the AV system 120. In an embodiment, the remote database 134 is embedded in the cloud computing environment 200 as described in Figure 2 The communication interface 140 transfers data collected from the sensors 121 or other data related to the operation of the AV 100 to the remote database 134. In an embodiment, the communication interface 140 transmits information related to teleoperation to the AV 100. In some embodiments, the AV 100 communicates with other remote (e.g., "cloud") servers 136.

[0078] In an embodiment, the remote database 134 also stores and transmits digital data (such as storing data on road and street locations). This data is stored in the memory 144 on the AV 100 or transferred to the AV 100 from the remote database 134 via a communication channel.

[0079] In an embodiment, the remote database 134 stores and transmits historical information (such as speed and acceleration distributions) related to the driving attributes of vehicles that previously traveled along the trajectory 198 at similar times during the day. In one implementation, such data can be stored in the memory 144 on the AV 100 or transferred to the AV 100 from the remote database 134 via a communication channel.

[0080] The computing device 146 located on the AV 100 algorithmically generates control actions based on both real-time sensor data and prior information, allowing the AV system 120 to perform its autonomous driving capabilities.

[0081] In an embodiment, the AV system 120 includes computer peripherals 132 coupled to the computing device 146 for providing information and alerts to the user of the AV 100 (e.g., an occupant or a remote user) and receiving input from the user. In an embodiment, the peripheral 132 is similar to the display 312, input device 314, and cursor controller 316 discussed below with reference to Figure 3 discussion. The coupling is wireless or wired. Any two or more of the interface devices may be integrated into a single device.

[0082] In an embodiment, the AV system 120 receives and enforces a privacy level of the occupant, such as specified by the occupant or stored in a profile associated with the occupant. The privacy level of the occupant determines how the use of specific information associated with the occupant (e.g., occupant comfort data, biometric data, etc.) stored in the occupant profile and / or stored on the cloud server 136 and associated with the occupant profile is permitted. In an embodiment, the privacy level specifies specific information associated with the occupant that is to be deleted once the ride is completed. In an embodiment, the privacy level specifies specific information associated with the occupant and identifies one or more entities authorized to access the information. Examples of the specified entities authorized to access the information may include other AVs, third-party AV systems, or any entity that may potentially access the information.

[0083] The privacy level of the occupant may be specified at one or more granularity levels. In an embodiment, the privacy level identifies specific information to be stored or shared. In an embodiment, the privacy level applies to all information associated with the occupant such that the occupant may specify not to store or share her personal information. The specification of the entities authorized to access the specific information may also be specified at various granularity levels. The various sets of entities authorized to access the specific information may include, for example, other AVs, the cloud server 136, specific third-party AV systems, etc.

[0084] In an embodiment, the AV system 120 or the cloud server 136 determines whether the AV 100 or another entity may access certain information associated with the occupant. For example, a third-party AV system attempting to access occupant input related to a specific spatio-temporal location must obtain authorization, for example, from the AV system 120 or the cloud server 136, to access the information associated with the occupant. For example, the AV system 120 uses the specified privacy level of the occupant to determine whether the occupant input related to the spatio-temporal location may be presented to a third-party AV system, the AV 100, or another AV. This enables the privacy level of the occupant to specify which other entities are permitted to receive data related to the actions of the occupant or other data associated with the occupant.

[0085] Figure 2Exemplary "cloud" computing environment. Cloud computing is a service delivery model that enables convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services). In a typical cloud computing system, one or more large cloud data centers house the machines for delivering the services provided by the cloud. Now refer to Figure 2 , the cloud computing environment 200 includes cloud data centers 204a, 204b, and 204c interconnected by a cloud 202. The data centers 204a, 204b, and 204c provide cloud computing services to computer systems 206a, 206b, 206c, 206d, 206e, and 206f connected to the cloud 202.

[0086] The cloud computing environment 200 includes one or more cloud data centers. Generally speaking, a cloud data center (e.g., Figure 2 the cloud data center 204a shown in Figure 2 ) refers to the physical arrangement of servers that make up a cloud (e.g., Figure 3 the cloud 202 shown in

[0087] or a particular part of the cloud). For example, servers are physically arranged in rooms, groups, rows, and racks within a cloud data center. A cloud data center has one or more zones, which include one or more server rooms. Each room has one or more rows of servers, and each row includes one or more racks. Each rack includes one or more individual server nodes. In some implementations, the servers in a zone, room, rack, and / or row are arranged into groups based on the physical infrastructure requirements of the data center facility (including power, energy, heat, heat source, and / or other requirements). In an embodiment, a server node is similar to Figure 3 the computer system described in

[0087] . The data center 204a has many computing systems distributed across multiple racks.

[0088] The computing systems 206a-f or cloud computing service consumers are connected to the cloud 202 via network links and network adapters. In an embodiment, the computing systems 206a-f are implemented as various computing devices, such as servers, desktops, laptops, tablets, smartphones, Internet of Things (IoT) devices, AV (including cars, drones, shuttles, trains, buses, etc.), and consumer electronics. In an embodiment, the computing systems 206a-f are implemented in or as part of other systems.

[0089] Figure 3 Illustrate the computer system 300. In an implementation, the computer system 300 is a dedicated computing device. The dedicated computing device is hardwired to perform these techniques, or includes digital electronic devices such as one or more application specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs) that are persistently programmed to perform the above techniques, or may include one or more general purpose hardware processors that are programmed to perform these techniques according to program instructions in firmware, memory, other memory, or a combination. Such a dedicated computing device may also combine custom hardwired logic, ASICs, or FPGAs with custom programming to complete these techniques. In various embodiments, the dedicated computing device is a desktop computer system, a portable computer system, a handheld device, a network device, or any other device that includes hardwired and / or program logic to implement these techniques.

[0090] In an embodiment, the computer system 300 includes a bus 302 or other communication mechanism for conveying information, and a hardware processor 304 coupled to the bus 302 for processing information. The hardware processor 304 is, for example, a general purpose microprocessor. The computer system 300 also includes a main memory 306, such as random access memory (RAM) or other dynamic storage device, coupled to the bus 302 to store information and instructions for execution by the processor 304. In one implementation, the main memory 306 is used to store temporary variables or other intermediate information during the execution of instructions to be executed by the processor 304. When these instructions are stored in a non-transitory storage medium accessible by the processor 304, the computer system 300 becomes a dedicated machine that is customized to perform the operations specified in the instructions.

[0091] In an embodiment, the computer system 300 also includes a read only memory (ROM) 308 or other static storage device coupled to the bus 302 for storing static information and instructions for the processor 304. A storage device 310, such as a magnetic disk, optical disk, solid state drive, or three-dimensional cross-point memory, is provided and coupled to the bus 302 to store information and instructions.

[0092] In an embodiment, computer system 300 is coupled via bus 302 to a display 312, such as a cathode ray tube (CRT), liquid crystal display (LCD), plasma display, light emitting diode (LED) display, or organic light emitting diode (OLED) display for displaying information to a computer user. An input device 314, including alphanumeric and other keys, is coupled to bus 302 for communicating information and command selections to processor 304. Another type of user input device is a cursor control 316, such as a mouse, trackball, touch display, or cursor direction keys for communicating direction information and command selections to processor 304 and for controlling cursor movement on display 312. Such input devices typically have two degrees of freedom in two axes (a first axis (e.g., the x-axis) and a second axis (e.g., the y-axis)), which allow the device to specify a position on a plane.

[0093] According to one embodiment, the techniques herein are performed by computer system 300 in response to one or more sequences of one or more instructions contained in main memory 306 being executed by processor 304. These instructions are read into main memory 306 from another storage medium, such as storage device 310. Execution of the instruction sequences contained in main memory 306 causes processor 304 to perform the process steps described herein. In alternative embodiments, hardwired circuitry is used in place of, or in combination with, software instructions.

[0094] As used herein, the term "storage medium" refers to any non-transitory medium that stores data and / or instructions that cause a machine to operate in a particular fashion. Such storage medium includes non-volatile and / or volatile media. Non-volatile media includes, for example, optical disks, magnetic disks, solid state drives, or three-dimensional cross-point memories such as storage device 310. Volatile media includes dynamic memory, such as main memory 306. Common forms of storage media include, for example, floppy disk, flexible disk, hard disk, solid state drive, magnetic tape, or any other magnetic data storage medium, CD-ROM, any other optical data storage medium, any physical medium with patterns of holes, RAM, PROM, and EPROM, FLASH-EPROM, NV-RAM, or any other memory chip or cartridge.

[0095] Storage media is distinct from transmission media but may be used in combination with transmission media. Transmission media participates in the transfer of information between storage media. For example, transmission media includes coaxial cables, copper wire, and fiber optics, including the wires that comprise bus 302. Transmission media can also take the form of acoustic or light waves, such as acoustic or light waves generated during radio wave and infrared data communications.

[0096] In an embodiment, various forms of media are involved in carrying one or more sequences of one or more instructions to the processor 304 for execution. For example, these instructions are initially executed on a disk or solid state drive of a remote computer. The remote computer loads the instructions into its dynamic memory and sends the instructions over a telephone line using a modem. The local modem of the computer system 300 receives the data on the telephone line and converts the data into an infrared signal using an infrared transmitter. The infrared detector receives the data carried in the infrared signal, and appropriate circuitry places the data on the bus 302. The bus 302 carries the data to the main memory 306, and the processor 304 retrieves and executes the instructions from the main memory 306. The instructions received by the main memory 306 may optionally be stored on the storage device 310 before or after being executed by the processor 304.

[0097] The computer system 300 also includes a communication interface 318 coupled to the bus 302. The communication interface 318 provides two-way data communication coupled to a network link 320 connected to a local network 322. For example, the communication interface 318 is an Integrated Services Digital Network (ISDN) card, a cable modem, a satellite modem, or a modem for providing a data communication connection to a corresponding type of telephone line. As another example, the communication interface 318 is a Local Area Network (LAN) card for providing a data communication connection to a compatible LAN. In some implementations, a wireless link is also implemented. In any such implementation, the communication interface 318 sends and receives electrical, electromagnetic, or optical signals carrying digital data streams representing various types of information.

[0098] The network link 320 typically provides data communication to other data devices through one or more networks. For example, the network link 320 provides a connection to a main computer 324 or to a cloud data center or device operated by an Internet Service Provider (ISP) 326 through the local network 322. The ISP 326 in turn provides data communication services through the worldwide packet data communication network now commonly referred to as the "Internet" 328. Both the local network 322 and the Internet 328 use electrical, electromagnetic, or optical signals carrying digital data streams. The signals through the various networks and the signals on the network link 320 and through the communication interface 318 are example forms of transmission media, where these signals carry digital data into and out of the computer system 300. In an embodiment, the network 320 includes the aforementioned cloud 202 or a part of the cloud 202.

[0099] The computer system 300 sends messages and receives data including program code via one or more networks, network link 320, and communication interface 318. In an embodiment, the computer system 300 receives code for processing. The received code is executed by the processor 304 upon receipt, and / or stored in the storage device 310, or stored in other non-volatile storage devices for later execution.

[0100] AV architecture

[0101] Figure 4 Shows an example architecture 400 for an AV (e.g., Figure 1 the AV 100 shown). The architecture 400 includes a perception module 402 (sometimes referred to as a perception circuit), a planning module 404 (sometimes referred to as a planning circuit), a control module 406 (sometimes referred to as a control circuit), a localization module 408 (sometimes referred to as a localization circuit), and a database module 410 (sometimes referred to as a database circuit). Each module plays a role in the operation of the AV 100. Collectively, the modules 402, 404, 406, 408, and 410 can be Figure 1 part of the AV system 120 shown. In some embodiments, any of the modules 402, 404, 406, 408, and 410 is a combination of computer software (e.g., executable code stored on a computer-readable medium) and computer hardware (e.g., one or more microprocessors, microcontrollers, application-specific integrated circuits [ASICs], hardware memory devices, other types of integrated circuits, other types of computer hardware, or any or all combinations of these hardware). The modules 402, 404, 406, 408, and 410 are each sometimes referred to as a processing circuit (e.g., computer hardware, computer software, or a combination of the two). Any or all combinations of the modules 402, 404, 406, 408, and 410 are also examples of processing circuits.

[0102] In use, the planning module 404 receives data representing a destination 412 and determines data representing a trajectory 414 (sometimes referred to as a route) that the AV 100 can travel to reach (e.g., arrive at) the destination 412. To enable the planning module 404 to determine the data representing the trajectory 414, the planning module 404 receives data from the perception module 402, the localization module 408, and the database module 410.

[0103] The perception module 402 uses one or more sensors 121, also as shown, for example, Figure 1 to identify nearby physical objects. The objects are classified (e.g., grouped into types such as pedestrians, bicycles, cars, traffic signs, etc.), and a scene description including the classified objects 416 is provided to the planning module 404.

[0104] The planning module 404 also receives data representing the AV position 418 from the positioning module 408. The positioning module 408 determines the AV position by using data from the sensor 121 and data from the database module 410 (e.g., geographic data) to calculate the position. For example, the positioning module 408 uses data from a GNSS (Global Navigation Satellite System) sensor and geographic data to calculate the longitude and latitude of the AV. In an embodiment, the data used by the positioning module 408 includes a high-precision map with lane geometric attributes, a map describing road network connection attributes, a map describing the physical attributes of the lanes (such as traffic rate, traffic volume, the number of vehicles and bicycle lanes, lane width, lane traffic direction, or lane marking type and location, or a combination thereof), and a map describing the spatial locations of road features (such as intersections, traffic signs, or other types of driving signals). In an embodiment, the high-precision map is constructed by adding data to a low-precision map via automatic or manual annotation.

[0105] The control module 406 receives data representing the trajectory 414 and data representing the AV position 418, and operates the control functions 420a - 420c (e.g., steering, throttle, brake, ignition system) of the AV in a manner that will cause the AV 100 to travel along the trajectory 414 to the destination 412. For example, if the trajectory 414 includes a left turn, the control module 406 will operate the control functions 420a - 420c as follows: the steering angle of the steering function will cause the AV 100 to turn left, and the throttle and brake will cause the AV 100 to pause and wait for passing pedestrians or vehicles before making the turn.

[0106] AV Input

[0107] Figure 5 Shows examples of the inputs 502a - 502d (e.g., Figure 4 ) used by the perception module 402 ( Figure 1 such as the sensor 121 shown in Figure 1 ) and the outputs 504a - 504d (e.g., sensor data). One input 502a is a LiDAR (Light Detection and Ranging) system (e.g.,

[0108] Another input 502b is a RADAR (radio detection and ranging) system. RADAR is a technology that uses radio waves to obtain data related to nearby physical objects. RADAR can obtain data related to objects that are not within the line of sight of the LiDAR system. The RADAR system generates RADAR data as output 504b. For example, the RADAR data is one or more radio frequency electromagnetic signals for constructing a representation of the environment 190.

[0109] Another input 502c is a camera system. The camera system uses one or more cameras (e.g., a digital camera using an optical sensor such as a charge-coupled device [CCD]) to obtain information related to nearby physical objects. The camera system generates camera data as output 504c. The camera data typically takes the form of image data (e.g., data in an image data format such as RAW, JPEG, PNG, etc.). In some examples, the camera system has multiple independent cameras for, e.g., the purpose of stereoscopic imaging (stereo vision), which enables the camera system to perceive depth. Although the objects perceived by the camera system are described here as "nearby", this is relative to the AV. In some embodiments, the camera system is configured to "see" distant objects (e.g., objects up to 1 kilometer or more in front of the AV). Thus, in some embodiments, the camera system has features such as sensors and lenses optimized for perceiving distant objects.

[0110] Another input 502d is a traffic light detection (TLD) system. The TLD system uses one or more cameras to obtain information related to traffic lights, street signs, and other physical objects that provide visual navigation information. The TLD system generates TLD data as output 504d. The TLD data often takes the form of image data (e.g., data in an image data format such as RAW, JPEG, PNG, etc.). The TLD system differs from a system that includes a camera in that: the TLD system uses a camera with a wide field of view (e.g., using a wide-angle lens or a fish-eye lens) to obtain information related to as many physical objects that provide visual navigation information as possible, so that the AV 100 can access all the relevant navigation information provided by these objects. For example, the field of view of the TLD system is about 120 degrees or greater.

[0111] In some embodiments, sensor fusion techniques are used to combine the outputs 504a - 504d. Thus, the individual outputs 504a - 504d are provided to other systems of the AV 100 (e.g., provided to, such as Figure 4The planning module 404) shown, or a single combined output or multiple combined outputs of the same type (e.g., using the same combination technique or combining the same outputs or both) or a single combined output or multiple combined outputs of different types (e.g., using different respective combination techniques or combining different respective outputs or both) may be used to provide the combined output to other systems. In some embodiments, early fusion techniques are used. Early fusion techniques are characterized by combining the outputs before applying one or more data processing steps to the combined output. In some embodiments, late fusion techniques are used. Late fusion techniques are characterized by combining the outputs after applying one or more data processing steps to the individual outputs.

[0112] Figure 6 An example of a LiDAR system 602 is shown (e.g., Figure 5 the input 502a) shown. The LiDAR system 602 emits light 604a - 604c from a light emitter 606 (e.g., a laser emitter). The light emitted by the LiDAR system is typically not in the visible spectrum; for example, infrared light is often used. Some of the emitted light 604b encounters a physical object 608 (e.g., a vehicle) and is reflected back to the LiDAR system 602. (The light emitted from the LiDAR system typically does not penetrate a physical object, e.g., a physical object in solid form.) The LiDAR system 602 also has one or more light detectors 610 for detecting the reflected light. In an embodiment, one or more data processing systems associated with the LiDAR system generate an image 612 representing the field of view 614 of the LiDAR system. The image 612 includes information representing the boundaries 616 of the physical object 608. Thus, the image 612 is used to determine the boundaries 616 of one or more physical objects near the AV.

[0113] Figure 7 An example of the LiDAR system 602 in operation is shown. In the scenario shown in this figure, the AV 100 receives both the camera system output 504c in the form of an image 702 and the LiDAR system output 504a in the form of LiDAR data points 704. In use, the data processing system of the AV 100 compares the image 702 with the data points 704. In particular, the physical objects 706 identified in the image 702 are also identified in the data points 704. Thus, the AV 100 perceives the boundaries of the physical objects based on the contours and densities of the data points 704.

[0114] Figure 8 Additional details of the operation of the LiDAR system 602 are shown. As described above, the AV 100 detects the boundaries of physical objects based on the characteristics of the data points detected by the LiDAR system 602. As Figure 8As shown, a flat object such as the ground 802 will reflect the light 804a - 804d emitted from the LiDAR system 602 in a consistent manner. In other words, since the LiDAR system 602 emits light at consistent intervals, the ground 802 will reflect the light back to the LiDAR system 602 at the same consistent intervals. When the AV 100 is traveling on the ground 802 and there is nothing blocking the road, the LiDAR system 602 will continue to detect the light reflected by the next valid ground point 806. However, if the object 808 blocks the road, the light 804e - 804f emitted by the LiDAR system 602 will be reflected from the points 810a - 810b in a manner inconsistent with the expected consistent manner. Based on this information, the AV 100 can determine that there is an object 808.

[0115] Path Planning

[0116] Figure 9 Illustrates (e.g., as Figure 4 shown in) a block diagram 900 showing the relationship between the inputs and outputs of the planning module 404. Generally, the output of the planning module 404 is a route 902 from a starting point 904 (e.g., a source location or an initial location) to an ending point 906 (e.g., a destination or a final location). The route 902 is typically defined by one or more segments. For example, a segment refers to a distance to be traveled on at least a portion of a street, road, highway, lane, or other physical area suitable for vehicle travel. In some examples, e.g., if the AV 100 is an off - road capable vehicle such as a four - wheel drive (4WD) or all - wheel drive (AWD) car, SUV, or pickup truck, the route 902 includes "off - road" segments such as unpaved paths or open fields.

[0117] In addition to the route 902, the planning module also outputs lane - level route planning data 908. The lane - level route planning data 908 is used to drive through the segments of the route 902 based on the conditions of the segments at a particular time. For example, if the route 902 includes a multi - lane road, the lane - level route planning data 908 includes trajectory planning data 910, where the AV 100 can use the trajectory planning data 910 to select a lane from the multiple lanes, e.g., based on whether an exit is approaching, whether there are other vehicles in one or more of the multiple lanes, or other factors that change over a period of minutes or less. Similarly, in some implementations, the lane - level route planning data 908 includes speed constraints 912 specific to a segment of the route 902. For example, if the segment includes pedestrians or unexpected traffic, the speed constraint 912 can limit the AV 100 to a slower driving speed than expected, e.g., a speed based on the speed limit data for that segment.

[0118] In an embodiment, the inputs to the planning module 404 include (e.g., fromFigure 4 ) database data 914, current location data 916 (e.g., Figure 4 AV position 418 shown), (e.g., for Figure 4 Destination data 918 and object data 920 (e.g., such as the destination 412 shown) Figure 4 4 (see Fig. 4 ). In some embodiments, the database data 914 includes rules (also referred to as a rulebook) used in planning. The rules are specified using a formal language (e.g., using Boolean logic or linear temporal logic (LTL)). In any given situation encountered by the AV 100, at least some of these rules will apply to the situation. A rule applies to a given situation if it has a condition that is satisfied based on information available to the AV 100 (e.g., information about the surrounding environment). Rules can have priorities. For example, a rule of "if the road is a highway, move to the leftmost lane" can have a lower priority than "if the exit is within one mile, move to the rightmost lane."

[0119] Figure 10 In path planning (eg, by planning module 404 ( Figure 4 )) uses a directed graph 1000. In general, if Figure 10 A directed graph 1000 such as the one shown is used to determine a path between any origin 1002 and destination 1004. In the real world, the distance separating the origin 1002 and destination 1004 may be relatively large (e.g., in two different metropolitan areas), or may be relatively small (e.g., two intersections adjacent to a city block or two lanes of a multi-lane road).

[0120] In an embodiment, a directed graph 1000 has nodes 1006a - 1006d representing different locations that an AV 100 may occupy between a starting point 1002 and an ending point 1004. In some examples, for instance, when the starting point 1002 and the ending point 1004 represent different urban areas, the nodes 1006a - 1006d represent segments of a road. In some examples, for instance, when the starting point 1002 and the ending point 1004 represent different locations on the same road, the nodes 1006a - 1006d represent different positions on that road. Thus, the directed graph 1000 includes information at different granularity levels. In an embodiment, a directed graph with a high granularity is also a sub - graph of another directed graph with a larger scale. For example, most of the information in a directed graph where the starting point 1002 and the ending point 1004 are far apart (e.g., many miles apart) is at a low granularity, and the directed graph is based on stored data, but the directed graph also includes some high - granularity information for a portion of the physical locations within the field of view representing the AV 100 in the directed graph.

[0121] The nodes 1006a - 1006d are different from objects 1008a - 1008b that cannot overlap with the nodes. In an embodiment, at a low granularity, the objects 1008a - 1008b represent areas that a car cannot pass through, such as areas without streets or roads. At a high granularity, the objects 1008a - 1008b represent physical objects within the field of view of the AV 100, such as other cars, pedestrians, or other entities with which the AV 100 cannot share physical space. In an embodiment, some or all of the objects 1008a - 1008b are static objects (e.g., objects that do not change position, such as streetlights or utility poles, etc.) or dynamic objects (e.g., objects that can change position, such as pedestrians or other cars, etc.).

[0122] Nodes 1006a - 1006d are connected by edges 1010a - 1010c. If two nodes 1006a - 1006b are connected by edge 1010a, the AV 100 can travel between one node 1006a and another node 1006b, for example, without having to travel to an intermediate node before reaching the other node 1006b. (That is, the AV 100 travels between two physical locations represented by the corresponding nodes.) Edges 1010a - 1010c are generally two - way in the sense that the AV 100 can travel from the first node to the second node or from the second node to the first node. In an embodiment, edges 1010a - 1010c are one - way in the sense that the AV 100 can travel from the first node to the second node, yet the AV 100 cannot travel from the second node to the first node. Edges 1010a - 1010c are one - way in cases where they represent, for example, one - way streets, individual lanes of streets, roads, or highways, or other features that can only be traversed in one direction due to legal or map constraints.

[0123] In an embodiment, the planning module 404 uses the directed graph 1000 to identify a path 1012 composed of nodes and edges between the starting point 1002 and the ending point 1004.

[0124] Edges 1010a - 1010c have associated costs 1014a - 1014b. The costs 1014a - 1014b are values representing the resources that will be spent if the AV100 selects that edge. A typical resource is time. For example, if the physical distance represented by one edge 1010a is twice the physical distance represented by another edge 1010b, the associated cost 1014a of the first edge 1010a can be twice the associated cost 1014b of the second edge 1010b. Other factors affecting time include expected traffic, the number of intersections, speed limits, etc. Another typical resource is fuel economy. Two edges 1010a - 1010b can represent the same physical distance, but, for example, due to road conditions, expected weather, etc., one edge 1010a requires more fuel compared to the other edge 1010b.

[0125] When the planning module 404 identifies a path 1012 between the starting point 1002 and the ending point 1004, the planning module 404 generally selects a path optimized for cost, for example, a path having the minimum total cost when adding together the individual costs of the edges.

[0126] AV Control

[0127] Figure 11 Shown (e.g., as Figure 4Block diagram 1100 of the inputs and outputs of control module 406 (shown). The control module operates in accordance with controller 1102, which controller 1102 includes, for example: one or more processors similar to processor 304 (e.g., one or more computer processors such as a microprocessor or a microcontroller or both); short-term and / or long-term data storage devices similar to main memory 306, ROM 308, and storage device 310 (e.g., memory, random access memory, or flash memory or both); and instructions stored in the memory that, when executed (e.g., by one or more processors), perform the operations of controller 1102.

[0128] In an embodiment, controller 1102 receives data representing a desired output 1104. The desired output 1104 typically includes speed, such as rate and heading. The desired output 1104 can be based, for example, on data received from planning module 404 (shown). In accordance with the desired output 1104, controller 1102 generates data that can be used as throttle input 1106 and steering input 1108. The throttle input 1106 represents, for example, the magnitude of engaging the throttle (e.g., acceleration control) of AV 100 by engaging the steering pedal or engaging another throttle control to achieve the desired output 1104. In some examples, the throttle input 1106 also includes data that can be used to engage the brakes of AV 100 (e.g., deceleration control). The steering input 1108 represents the steering angle, e.g., the angle at which the steering control of the AV (e.g., the steering wheel, the steering angle actuator, or other functions for controlling the steering angle) should be positioned to achieve the desired output 1104. Figure 4 In an embodiment, controller 1102 receives feedback used in adjusting the inputs provided to the throttle and steering. For example, if AV 100 encounters a disturbance 1110 such as a hill, the measured rate 1112 of AV 100 drops below the desired output rate. In an embodiment, any measured output 1114 is provided to controller 1102 such that, for example, the required adjustment is made based on the difference 1113 between the measured rate and the desired output. The measured output 1114 includes measured position 1116, measured speed 1118 (including rate and heading), measured acceleration 1120, and other outputs measurable by the sensors of AV 100.

[0129]

[0130] ​In an embodiment, information related to the disturbance 1110 is pre-detected by a sensor such as a camera or a LiDAR sensor, for example, and this information is provided to the predictive feedback module 1122. Then, the predictive feedback module 1122 provides information that the controller 1102 can use to adjust accordingly to the controller 1102. For example, if the sensors of the AV 100 detect (“see”) a hill, the controller 1102 can use this information to prepare to engage the throttle at an appropriate time to avoid significant deceleration.

[0131] Figure 12 Block diagram 1200 showing the inputs, outputs, and components of the controller 1102. The controller 1102 has a rate analyzer 1202 that affects the operation of the throttle / brake controller 1204. For example, the rate analyzer 1202 instructs the throttle / brake controller 1204 to accelerate or decelerate using the throttle / brake 1206 based on feedback received by the controller 1102 and processed by the rate analyzer 1202, for example.

[0132] The controller 1102 also has a lateral tracking controller 1208 that affects the operation of the steering wheel controller 1210. For example, the lateral tracking controller 1208 instructs the steering wheel controller 1210 to adjust the position of the steering angle actuator 1212 based on feedback received by the controller 1102 and processed by the lateral tracking controller 1208, for example.

[0133] The controller 1102 receives several inputs for determining how to control the throttle / brake 1206 and the steering angle actuator 1212. The planning module 404 provides information that the controller 1102 uses, for example, to select the heading when the AV 100 starts operating and to determine which road segment to cross when the AV 100 reaches an intersection. The localization module 408 provides information describing the current location of the AV 100 to the controller 1102, for example, so that the controller 1102 can determine whether the AV 100 is at the location expected based on the way the throttle / brake 1206 and the steering angle actuator 1212 are being controlled. In an embodiment, the controller 1102 receives information from other inputs 1214, such as information received from a database, a computer network, etc.

[0134] Autonomous vehicle performance evaluation system

[0135] The computerized performance evaluation system can be configured to evaluate the behavior of an AV during autonomous operation. For example, an AV (e.g., AV 100, as referenced Figures 1-12The (described) can be configured to perform autonomous navigation operations, such as autonomously determining a path from one location to another location and traversing the path, etc. In addition, a performance evaluation system can select a set of scenarios to test the safety, efficiency, and effectiveness of the AV when performing these operations. In some implementations, the performance evaluation system can use one or more simulated scenarios to perform at least some of these tests to test the behavior of the AV in a computerized virtual environment. In some implementations, one or more real-world scenarios can be used to perform at least some of these tests to test the actual behavior of the AV in a physical environment.

[0136] In addition, the AV can be modified and / or reconfigured based on the tests to improve the performance of the AV. For example, one or more components of the AV, the configuration of these components, and / or the rulebook of the AV can be modified based on the tests such that the AV performs more safely, efficiently, and / or effectively when performing autonomous operations. As an example, if the tests indicate that the AV behaves in an unsafe or other undesirable manner under certain conditions, the AV can be modified and / or reconfigured to reduce the likelihood that the AV will behave in that manner in the future under these conditions.

[0137] In some implementations, the performance evaluation system can be used as part of an iterative prototyping and validation process for developing the AV. For example, the configuration of the AV can be alternately verified and modified multiple times continuously to progressively identify and correct unsafe or other undesirable behaviors of the AV.

[0138] Figure 13 An example system 1300 for iteratively prototyping an AV and validating the performance of the prototyped AV is shown. System 1300 includes an AV prototyping system 1302 and an AV performance evaluation system 1304.

[0139] The AV prototyping system 1302 can be implemented at least in part using one or more computer systems (e.g., as Figures 1-3 shown, cloud server 136, computing environment 200, and / or computer system 300). During an example operation of the AV prototyping system 1302, the AV prototyping system 1302 determines the configuration of the AV. For example, the AV prototyping system 1302 can determine one or more rulebooks 1306, a hardware configuration 1308, and / or a software configuration 1310 for the AV.

[0140] As described above (e.g., with reference to Figure 4 and Figure 9), the rule book 1306 includes one or more rules used by a planning module (e.g., planning module 404) to plan the path of the AV. For example, given a particular situation encountered by the AV, at least some of the rules in the rule book may apply to that situation. The AV can autonomously perform one or more actions based on the applicable rules. For example, as described above (e.g., referring to Figure 11 ), the AV can steer, apply throttle, and / or brake in a particular manner according to the rules.

[0141] As an example, the rule book can include at least one traffic rule. For example, the traffic rule can specify that the AV obeys traffic signals, drives with the traffic flow, and maintains a legal speed under some or all conditions.

[0142] As another example, the rule book can include at least one safety rule. For example, the safety rule can specify that the AV avoids collisions with other objects, maintains a safe distance between other vehicles and pedestrians, and performs evasive maneuvers to avoid collisions or other unsafe situations.

[0143] As another example, the rule book can include at least one occupant comfort rule. For example, the occupant comfort rule can specify that the AV accelerates and decelerates smoothly and turns smoothly.

[0144] As another example, the rule book can include at least one vehicle performance rule. For example, the vehicle performance rule can specify that the AV maintains certain speed, braking, acceleration, and turning limits of the vehicle.

[0145] In some implementations, the rule book 1306 can also be used to evaluate the performance of the AV. For example, the rule book 1306 can define a particular expected behavior of the AV (e.g., a combination of one or more traffic rules, safety rules, occupant comfort rules, vehicle performance rules, and / or any other rules that specify the expected behavior of the AV). Violating the rules in the rule book 1306 can result in an increase in a violation metric (e.g., a numerical score indicating the frequency and / or severity of the AV's undesirable operations). Although the various rules of the rule book are illustrated above, these are merely illustrative examples. In practice, instead of or in addition to the above rules, the rule book can include one or more additional rules that specify the particular behavior of the AV in a particular situation.

[0146] The hardware configuration 1308 of the AV can include any arrangement of the physical components of the AV described herein. For example, a particular hardware configuration 1308 can specify that the AV includes a certain number of different types of physical components described herein (e.g., see Figures 1-12For each physical component in (), a specific location and / or orientation is specified for each physical component among these physical components on the AV, a specific physical arrangement of each component and its components is specified, and a specific interconnection (e.g., electrical and / or communication interconnection) between these physical components and parts is specified. As another example, the specific hardware configuration 1308 can also specify that the AV has certain physical characteristics (e.g., mass, dimensions, shape, center of gravity, etc.). As another example, the specific hardware configuration 1308 can also specify that the AV has certain performance characteristics (e.g., maximum speed, acceleration ability, braking compatibility, turning ability, etc.).

[0147] The software configuration 1310 of the AV can include any arrangement of the software components described herein. For example, a specific software configuration 1310 can specify that the AV includes a specific arrangement of software modules, each software module being configured to perform the above-described software-based operations (e.g., referring to Figures 1-12 ) of certain sets. As another example, the specific software configuration 1310 can also specify that the software components of the AV retrieve certain types of data from one or more physical components and / or other software components, process the data in a specific manner, and transmit the processed data to other physical and / or software components.

[0148] In some implementations, the AV prototyping system 1302 can automatically (e.g., without input from a user) determine the configuration of the AV. For example, the AV prototyping system 1302 can use one or more machine learning and / or unsupervised learning processes to determine a set of rulebooks 1306, a hardware configuration 1308, and a software configuration 1310 of the AV.

[0149] In some implementations, the AV prototyping system 1302 can determine the configuration of the AV at least in part based on input from one or more users (e.g., one or more human designers). For example, a user can specify one or more rulebooks 1306, a hardware configuration 1308, and / or a software configuration 1310 of the AV. The AV prototyping system 1302 can determine the configuration of the AV based on the user's specification.

[0150] When determining a specific configuration of the AV, the AV prototyping system 1302 provides this configuration to the AV performance evaluation system 1304 for testing. As described above, the AV performance evaluation system 1304 can test the AV (with the configuration specified by the AV prototyping system 1302) in multiple scenarios 1312 to evaluate the behavior of this configuration of the AV under various conditions.

[0151] In some implementations, each scenario 1302 can specify a specific combination of roads, obstacles, other vehicles, pedestrians, and traffic flow that the AV has to cross. For example, Figure 14A and 14BShows a simplified example of scenario 1312 for testing the performance of an AV. As Figure 14A shown, scenario 1312 includes a network of interconnected roads 1402 that the AV can traverse, with each interconnected road 1402 having a specific traffic flow (e.g., traffic in a specific direction in each lane of the road). Additionally, as Figure 14B shown, scenario 1312 includes several other vehicles 1404 traveling along these roads 1402. For example, vehicles 1404 can be located at certain locations on roads 1402 and can be traveling along these roads 1402 at a specific speed and direction. Additionally, as Figure 14B shown, scenario 1312 includes a designated goal 1406 for the AV. The goal can be, for example, a specific task or series of tasks that the AV is to perform in scenario 1302. For example, in the Figure 14B example shown, the goal 1406 can be a right turn from road 1402a to intersecting road 1402b.

[0152] As described above, in some implementations, at least some of the scenarios 1302 can be simulated scenarios. For example, at least some of the scenarios 1302 can specify a computerized virtual environment for testing the AV, the computerized virtual environment including one or more virtual roads, obstacles, other vehicles, pedestrians, traffic flows, and environmental conditions that the AV is to traverse. Additionally, at least some of the scenarios 1302 can specify goals for the AV to perform within the virtual environment. Additionally, given the configurations specified by the AV prototyping system 1302, computerized simulations can be used to predict the behavior of the AV within the virtual environment. In some implementations, the computerized simulations can be performed at least in part by the AV performance evaluation system 1304.

[0153] As described above, in some implementations, at least some of the scenarios 1302 can be real-world scenarios. For example, at least some of the scenarios 1302 can specify a physical environment for testing the AV. The AV can be physically deployed into the environment and instructed to perform autonomous operations within the environment. In some implementations, the results of the tests can be provided to the AV performance evaluation system 1304. For example, the results of the tests can be manually input by a user (e.g., a human observer) and / or automatically input by a computer system that observed the tests.

[0154] The AV performance evaluation system 1304 outputs one or more performance metrics for the AV based on the tests.

[0155] In some implementations, the performance evaluation system 1304 can output metrics indicating the number of times the AV violates legal, safety, and / or comfort constraints in each scenario and the severity of each of these violations.

[0156] As an example, these metrics can indicate the number of times the AV contacts another object or a pedestrian on the road. Additionally, the metrics can classify the contact based on severity. For example, contacting a pedestrian can be considered more severe than contacting a curb.

[0157] As another example, the metrics can indicate the number of times the AV violates certain traffic rules or regulations (such as exceeding the speed limit of the road, violating traffic signals, or driving against the traffic flow of the road, etc.). The metrics can also classify the violation contacts based on severity. For example, exceeding the speed limit to a lesser extent can be considered less severe than exceeding the speed limit to a greater extent.

[0158] As another example, the metrics can indicate the number of times the AV behaves in a way that will cause discomfort to the occupants (such as accelerating, braking, and / or changing the direction of the AV by more than certain comfort limits, etc.). The metrics can also classify the discomfort based on severity. For example, exceeding the acceleration limit to a lesser extent can be considered less severe than exceeding the acceleration limit to a greater extent.

[0159] As another example, the metrics can indicate the number of times the AV deviates from a previously planned route and the degree of these deviations (e.g., represented by the added distance to the route and / or added travel time).

[0160] As another example, the metrics can indicate the number of times the AV exceeds the performance limits of the AV and / or is at risk of exceeding the performance limits of the AV. For example, the metrics can indicate the number of times the AV accelerates, brakes, and / or turns in a way that will exceed the safety or design envelope of the AV and / or is at risk of exceeding the safety or design envelope of the AV. The metrics can also indicate the severity of these deviations. For example, the metrics can indicate the extent to which the AV exceeds the acceleration, braking, and steering capabilities of the AV, and / or how much the AV exceeds these capabilities.

[0161] As another example, the AV performance evaluation system 1304 can output metrics indicating the amount of time required for the AV to complete various scenarios. As another example, the AV performance evaluation system 1304 can output metrics indicating the average speed of the AV during various scenarios. As another example, the AV performance evaluation system 1304 can output metrics indicating the amount of resources (e.g., fuel and / or electricity) consumed by the AV during various scenarios.

[0162] The AV performance evaluation system 1304 provides one or more performance metrics to the AV prototyping system 1302. Based on the one or more performance metrics, the AV prototyping system 1302 can modify and / or reconfigure the AV to improve the performance of the AV. For example, the AV prototyping system 1302 can modify the rulebook 1306, hardware configuration 1308, and / or software configuration 1310 of the AV so that the AV can operate autonomously more safely, more efficiently, and / or more effectively.

[0163] The AV performance evaluation system 1304 (e.g., as described above) can be used to evaluate the modified configuration of the AV. This process can be performed multiple times in succession to iteratively improve the performance of the AV through successive incremental modifications and / or reconfigurations of the AV.

[0164] In some cases, given a specific resource budget, the AV performance evaluation system 1304 can select a set of scenarios 1312 to maximize the amount of information obtained regarding the performance of the AV. For example, the performance of the AV can be represented as a random variable with a specific probability distribution. Certain scenarios 1312 in which the entropy of the random variable decreases by a specific amount when used to evaluate the performance of the AV can be preferentially selected over other scenarios 1312 in which the entropy of the random variable decreases by a relatively small amount. Additionally, the selected set of scenarios 1312 can be included in a set of standard tests to evaluate the performance of the AV during the development and validation of the AV's autonomous navigation system.

[0165] Figure 15 An example process 1500 for selecting a set of scenarios for testing the performance of an AV is shown. Process 1500 can be performed, for example, by the AV performance evaluation system 1304 to select a set of scenarios 1312 from a candidate scenario pool to test the performance of the AV.

[0166] According to process 1500, a candidate scenario pool is identified. Additionally, each of these scenarios is mapped to one or more vectors in a scenario space (block 1502). As an example, the various characteristics of a candidate scenario can correspond to different dimensions of the scenario space. Vectors in the scenario space can be used to represent the presence or absence of that characteristic in a particular candidate scenario.

[0167] For example, at least some of the candidate scenarios can include a pedestrian crossing in front of the AV. Candidate scenarios with this characteristic can be represented by vectors each including a specific value (e.g., the value "1") in the first dimension of the scenario space. Candidate scenarios without this characteristic can be represented by vectors each including another value (e.g., the value "0") in the first dimension of the scenario space.

[0168] As another example, at least some of the candidate scenarios can include the presence of other vehicles traveling in the same lane as the AV. Candidate scenarios with this characteristic can be represented by vectors each including a specific value (e.g., the value "1") in the second dimension of the scenario space. Candidate scenarios without this characteristic can be represented by vectors each including another value (e.g., the value "0") in the second dimension of the scenario space.

[0169] As another example, at least some of the candidate scenarios can include the presence of other vehicles traveling in another lane different from the AV. Candidate scenarios with this feature can be represented by vectors each including a specific value (e.g., the value "1") in the third dimension of the scenario space. Candidate scenarios without this feature can be represented by vectors each including another value (e.g., the value "0") in the third dimension of the scenario space.

[0170] As another example, at least some of the candidate scenarios can include vehicles traveling in the same direction as the AV. Candidate scenarios with this feature can be represented by vectors each including a specific value (e.g., the value "1") in the fourth dimension of the scenario space. Candidate scenarios without this feature can be represented by vectors each including another value (e.g., the value "0") in the fourth dimension of the scenario space.

[0171] As another example, at least some of the candidate scenarios can include vehicles traveling in the opposite direction to the AV. Candidate scenarios with this feature can be represented by vectors each including a specific value (e.g., the value "1") in the fifth dimension of the scenario space. Candidate scenarios without this feature can be represented by vectors each including another value (e.g., the value "0") in the fifth dimension of the scenario space.

[0172] Although the above illustrates example features and dimensions, these are merely illustrative examples. In practice, the vector space can include any number of dimensions representing any number of different features of the candidate scenarios.

[0173] In addition, the performance of the AV is determined in the scenario space (block 1504). For example, using the performance evaluation system 1304, the performance of the AV can be tested in some or all of the candidate scenarios. One or more performance metrics (e.g., as described above) can be used to represent the results of the test.

[0174] In addition, for each of the tested scenarios, the amount of information obtained regarding the performance of the AV is determined (block 1506). In some implementations, the performance of the AV can be represented as a random variable with a specific probability distribution. The amount of information obtained can refer to a specific degree of entropy reduction in that variable and / or be related to that specific degree of entropy reduction.

[0175] In addition, a set of scenarios is selected from the candidate scenarios based on the amount of information obtained through each of the tested scenarios (block 1508). As an example, certain candidate scenarios that reduce the entropy of the random variable by a specific degree can be preferentially selected relative to other candidate scenarios that reduce the entropy of the random variable by a relatively small degree. The selected set of scenarios can be included in a set of standard tests to evaluate the performance of the AV during the development and validation of the AV's autonomous navigation system.

[0176] This process can be beneficial in mitigating the effects of diminishing returns when testing the performance of an AV. For example, Figure 16 A simplified relationship between the number of scenarios tested (horizontal axis) and the total amount of information obtained regarding the performance of the AV (vertical axis) is shown and is represented as curve 1602. As the number of scenarios tested (e.g., from left to right) increases, the total information related to the AV performance (e.g., from bottom to top) also increases. However, the marginal amount of information obtained decreases with each additional test, which is represented by the gradually decreasing slope of curve 1602 from left to right. When the marginal amount of information obtained is low enough (e.g., less than a specific threshold amount), the test can be interrupted. Thus, the benefits of conducting each additional test can be balanced against the resource costs associated with conducting the test.

[0177] In some implementations, this process enables the performance of an AV to be tested faster and more efficiently. For example, a limited set of scenarios can be selected for testing such that each scenario in the set provides useful information for evaluating the performance of the AV. Additionally, as described above, the set can be selected to mitigate the effects of diminishing returns. Thus, a smaller number of test scenarios (e.g., compared to the number of test scenarios that might be employed in a brute-force approach) can be used to determine the performance of the AV.

[0178] In some implementations, the threshold for determining when to interrupt the test can be selected empirically. For example, the threshold can be an adjustable value that enables a user to balance the desire to obtain information regarding the performance of the AV against the resource consumption for obtaining that information. For example, if a greater amount of information regarding the performance of the AV is desired, the threshold can be set to a relatively low value (e.g., such that a larger number of scenarios are used to conduct a larger number of tests until the threshold is reached), even though each additional test conducted will consume a greater amount of resources. As another example, if resources are desired to be consumed in a more conservative manner, the threshold can be set to a relatively high value (e.g., such that fewer scenarios are used to conduct fewer tests until the threshold is reached).

[0179] In some implementations, the number of scenarios selected for testing can be constrained based on a budget metric. For example, each candidate scenario can have an associated resource cost for using the candidate scenario to conduct a test. Candidate scenarios can be selected such that the resource cost of the selected scenarios does not exceed a specific budget. As an example, if the resource cost of each candidate scenario is one unit and the total budget is no greater than 10 units, up to 10 candidate scenarios can be selected for testing. This can be useful, for example, in restricting the resources that will be consumed when testing the performance of an AV. In some implementations, the budget can be selected empirically.

[0180] In some implementations, candidate scenarios can be grouped into multiple distinct groups based on similarities and / or differences between the candidate scenarios. Additionally, candidate scenarios can be selected for testing based on these groups.

[0181] As described above, each candidate scenario can be represented by a corresponding vector in the scenario space. Based on similarities and / or differences between the vectors, the candidate scenarios can be clustered into one or more groups. Generally, candidate scenarios within a common group can be more similar to each other, while candidate scenarios in different groups can be less similar to each other. Candidate scenarios can be selected from the groups in a way that maximizes (or otherwise enhances) the amount of information that will be obtained by testing with the selected scenarios. In some implementations, candidate scenarios can be selected from the groups according to a greedy algorithm (e.g., a heuristic for problem-solving that makes a local visual selection at each of several stages of the selection process).

[0182] In some implementations, candidate scenarios can be grouped according to a hierarchy. For example, the levels of the hierarchy can represent the relative similarities between the nodes of the hierarchy. Nodes that share a common node at a higher level of the hierarchy (e.g., closer to the "root" of the hierarchy) can indicate that these nodes have a lesser degree of similarity to each other. Conversely, nodes that share a common node at a lower level of the hierarchy can indicate that these nodes have a greater degree of similarity to each other. Candidate scenarios can be selected from the hierarchy in a way that maximizes (or otherwise enhances) the amount of information that will be obtained by testing with the selected scenarios. In some implementations, candidate scenarios can be selected from the hierarchy according to a greedy algorithm. In some implementations, the hierarchy can be a Bayesian hierarchical model. In some implementations, candidate scenarios can be grouped according to other graphical representations such as Bayesian networks, factor graphs, hidden Markov models (HMMs), or any other structure.

[0183] Example techniques for selecting scenarios based on information gain

[0184] As described above, the AV performance evaluation system 1304 can select scenarios for testing the behavior of the AV based on the amount of information that will be obtained by testing each scenario. The following describes example techniques for selecting scenarios based on information gain.

[0185] Generally, a hierarchical statistical model can be used to represent AV performance. Additionally, a set of scenarios can be selected to evaluate this performance across an operational design domain (ODD). The hierarchy can support information reuse from one ODD to the next, and since it is not used directly, it can reduce the sensitivity of performance estimation to event frequencies. Additionally, criteria can be used to evaluate the value of testing a particular scenario, and the example sampling techniques described herein can provide approximate optimality with respect to this criteria.

[0186] Typically, as summarized in Figure 17 , the selection process can include two steps. First, AV performance is modeled as a Bayesian hierarchical model. The formal rules of expected AV behavior and the occurrence of additional violation metrics facilitate the quantitative assessment of AV performance in any scenario. The factors constituting the ODD (such as the area of operation, etc.) can be used as levels in the hierarchy. In some implementations, the area can be a specific town, city, county, state, country, or some parts thereof. This representation uses the ODD factors as analysis units that each have only a few observations. In practice, it is difficult to observe scenarios corresponding to an ODD where all factors are fixed, such as a blue stop sign with a specific road curvature and high rain density and low communication reliability. The number of qualified observations decreases with the amount of fixed factors. In some implementations, this part of the process can be referred to as a flat, as each level of the hierarchy can effectively form a hyperplane.

[0187] Second, inferences about AV performance can be made through this Bayesian hierarchical model. This allows for the calculation of the information gain on the system provided by each new scenario, which can be used as a metric to evaluate the relevance of new scenarios and sets of scenarios. The Bayesian hierarchical model provides the conditional independence property between AV performances in different scenario space hyperplanes, which makes the information gain submodular. This means that the information gain has the property of diminishing returns. For example, as more scenarios are added, the increase in information becomes negligible. Therefore, a greedy algorithm for submodular optimization can be used to provide a set of scenarios to provide an information gain (sample) close to the maximum level. A stopping criterion (e.g., for interrupting the selection of additional scenarios for a set of scenarios) can be defined such that when considering the statistical confidence level, the information gain no longer increases, the selection is interrupted.

[0188] To demonstrate the effectiveness of this process, experiments were conducted by simulating AVs on the 2020 CARLA challenge (CARLA team) routes. The results of this experiment show a stable level of information gain with 90% confidence after testing only about 7% of the scenario space, while another technique (Latin hypercube sampling) explores 33% more scenarios to obtain the same amount of information.

[0189] As described below, relying on the quantitative evaluation of behavioral rules, a Bayesian hierarchical model can be used to present AV performance.

[0190] In addition, information gain can be used as a metric to quantify the relevance of new scenarios or a set of scenarios.

[0191] In addition, a greedy algorithm for submodular optimization can be used as part of the selection algorithm (e.g., to provide an approximate optimality guarantee).

[0192] In addition, a stopping criterion can be used to constrain the size of the set of scenarios. Specifically, using a given statistical confidence level, the selection of additional scenarios for the set of scenarios can be interrupted when the information gain remains at a stable level.

[0193] Example Processing

[0194] A. Plane

[0195] Let be the scenario space from which to sample. A scenario S ∈ V contains information related to what happens in the environment external to the system under test. It can be assumed that S can be characterized as an element of, where n is the number of features in the environment. These features can be based on scenario ontologies or industry recommendations defined by ODD. For a more targeted search, these coordinates can also correspond to the parameters of a specific logical scenario. The scenarios under consideration can be generated in a simulation or a real-world setting. Given n features for describing a scenario, such as cloud density, the presence of a construction site, or the number of dynamic agents, and k potential values for each feature, then for 50 features each of which can take 6 values, this represents approximately 10 10 scenarios. Although simulation allows testing of the AV at a fraction of the cost of trace testing, generating such a large number of scenarios may still require a large amount of computing power.

[0196] In addition, the AV performance can be in any given scenario. The rulebook framework provides rules associated with violation metrics to evaluate the AV performance in a scenario according to explicit behavioral specifications. In fact, the AV performance may be affected by many different concepts such as traffic laws, polite driving manners, or ride comfort. And even though this example experiment uses the number of collisions, the technique is compatible with using any performance metric. Once a specific rule is converted into a formal logical statement, any trajectory in any scenario can be evaluated and a violation score can be given.

[0197] The performance of the AV in the scenario space can be defined as where is a random variable. The goal is to sample the scenarios that will provide the most information about the distribution of . Therefore, some structure may need to be established after the distribution of . In fact, if the performance is completely chaotic, there will be no relationship between rule violations in two different scenarios, nothing learned from one scenario can be generalized to another scenario, and it may be necessary to test every scenario.

[0198] In this example, it is assumed that a Bayesian hierarchical model can be fitted to the observable AV performance. Figure 18Shows a representation of the Bayesian hierarchical model 1800. The lightly shaded nodes 1802a - 1802d represent unobservable variables, and the darkly shaded nodes 1804a - 1804e represent the random variables of interest.

[0199] Since the parameters are sampled from a probability distribution, lowercase letters are used to specify both the random variables for the parameters and hyperparameters and their realizations. The hyperplane of the scenario space is defined as a set of scenarios where a characteristic such as the town of operation is fixed. Thus, the hypothesis can be reformulated as follows: For the p hyperplanes of the scenario space, it can be assumed that there exist b1,..., b p and σ such that:

[0200] X i |b p ,σ ∼ P(x i |b p ,σ) (Equation 1)

[0201] b p |σ ∼ P(b p |σ)

[0202] σ ∼ P(σ)

[0203] where: σ is the hyperparameter with hyperprior P(σ), b1,..., b p are generated from a population with a distribution governed by the hyperparameter σ, and P(x i |b p ,σ) is the likelihood of the AV performance, where P(b p ,σ) serves as its prior distribution. Sufficient priors and hyperpriors can be chosen as part of the model fitting process.

[0204] The interpretation of this model is as follows: Assume that the AV performance follows the distribution of the parameter b p in the hyperplane p. This distribution can be inferred from data observations shown in the experimental and results section below, or from expert knowledge. When tested in the scenarios that occur in the town p, the sample x i can be effectively obtained from this distribution. In different towns k, the shape of the distribution of the AV performance will be the same, but the parameter b k will likely be different because b k , b p itself is sampled from P(σ). For example, the number of vehicle permit violations by the AV in a certain scenario can follow a lognormal distribution, but in a dense town with more traffic, the mean may be higher compared to a town with larger roads and fewer vehicles.

[0205] The quality of the set of scenarios determined by this example technique depends on the quality of the fit of the Bayesian hierarchical model to the AV performance. Poor fit of the parameters and hyperparameters will result in poor estimation of the information gain, so the algorithm will be optimized for the wrong values. However, there are techniques to ensure the quality of the fit of the Bayesian hierarchical model, such as posterior predictive checks, etc.

[0206] B. Samples

[0207] This section focuses on how to use this AV performance representation for sampling.

[0208] The following definition of entropy can be used to represent the uncertainty around the intrinsic performance parameter σ:

[0209] H(σ) = -∑ σ P(σ) log P(σ) (Equation 2).

[0210] If is the set of observed scenarios, then the information gain related to σ obtained from the performance in the observation is defined as:

[0211]

[0212] In this scenario space, there are sets of scenarios of size a, where a ≤ k n . The goal is to examine these options and select the set that maximizes the information gain before generating scenarios in the simulation engine or closed-loop testing. Therefore, the optimization problem is defined as:

[0213] Constraints: (Equation 4),

[0214] where: C is the scenario budget, and f can be for the overall AV evaluation across the entire ODD or for the evaluation of the AV performance in the hyperplane of the ODD such as a specific town C is the equivalent of the stopping criterion. The conditional entropy can only be approximated to a certain value with a given confidence interval. When the approximated information gain stops increasing in a statistically significant manner, the optimization can be stopped, which determines C.

[0215] Even in a simple setting where all scenarios have the same cost, this optimization problem is NP-hard. However, both potential objective functions are monotone submodular. A function f: is submodular if and only if for each and e ∈ V,

[0216] f(A ∪ {e}) - f(A) ≥ f(B ∪ {e}) - f(B) (Equation 5).

[0217] This means that adding scenarios to an already large set will provide a lower information gain compared to adding scenarios to a smaller test set. This reflects the intuition that adding more and more scenarios to a set will result in diminishing returns despite positive gains. For example, as Figure 16 shown, after adding more and more discrete elements (e.g., scenarios) to the set that defines the function, the function increases by smaller and smaller amounts.

[0218] This property appears here because, given the hyperparameter σ, the AV performance in different scenarios is conditionally independent. In general, the information gain regarding a certain random variable obtained by revealing another dependent random variable is not submodular. However, for a fixed AV stack, there may exist a hyperplane in the scenario space where, given this hyperplane, the AV performance in one scenario is independent of the AV performance in another scenario. In the experiments (detailed below), the number of collisions for each scenario follows a Poisson distribution with a fixed mean in a given town. No additional assumptions about the AV performance are required for the submodularity property to hold true.

[0219] Since f is submodular, a greedy algorithm can be used to provide an approximately optimal solution to Equation 4. Starting from an empty set of scenarios, this heuristic selects the next scenario with the highest information gain at each iteration.

[0220]

[0221] Since this selection problem is NP-hard, no algorithm can find a solution in polynomial time. However, the greedy algorithm can find a solution within a factor of the optimal value. Specifically, if the maximum information gain for C scenarios is 1, a set of scenarios of size C can be obtained that yields an information gain of at least 0.63.

[0222] Experiments and Results

[0223] Experiments were conducted to demonstrate the use of this experiment processed using autonomous vehicle logs obtained from computerized AV simulations (CARLA).

[0224] A. Data Generation

[0225] In this experiment, the model of the autonomous vehicle available in CARLA was also used to simulate background traffic in the simulation. The vehicle attempted to follow routes in six different towns for a total of 132 scenarios with a varying number of traffic participants. These 3 features are the coordinates of the scenarios shown in Table 1.

[0226]

[0227] Table 1. Scenario Definitions

[0228] Figure 19 Maps 1900a and 1900b showing two of the towns under consideration. As Figure 19 shown, town 03 (represented by map 1900a) substantially has more intersections and roundabouts, while town 06 (represented by map 1900b) has a long stretch of straight road.

[0229] Record the number of collisions involving AVs in each scenario, and this number of collisions is chosen as the performance metric X V .

[0230] The Poisson distribution is commonly used to represent the probability of multiple events occurring independently over a fixed interval. Figure 20 Show X V in all scenarios, the probability distribution 2000 of which aligns well with the Poisson hypothesis. It is worth noting that since the AV implementation used is typically only for background traffic and only consumes map and route information, this AV implementation encounters a large number of collisions for each scenario due to the lack of a perception module. Therefore, this AV implementation does not react to dynamic objects in the scene. However, the information related to the AV implementation is not necessary for applying this technology.

[0231] B. Bayesian Hierarchical Model

[0232] After data analysis, it is determined that the town is the scenario coordinate that has the greatest impact on the number of collisions in each scenario. For example, Figure 21 shows the collision distributions 2100a - 2100f along the route for six different towns in CARLA. This grouping can be inferred from the small amount of data already available or can be decided based on expert knowledge.

[0233] Therefore, the scenario space is cut by the town and the following Bayesian hierarchical model is fitted:[[]]

[0234]

[0235] For parameter b p and hyperparameter σ, a half - normal distribution is chosen as the weakly informative prior. PyMC3 (PyMC3 development team) is used to perform this fitting. The results of the fitted Bayesian hierarchical model are shown in Figure 22A (e.g., showing the posterior distribution of unobservable parameters). The average number of collisions for each scenario thus obtained varies widely between less than 1 for some towns and greater than 2.5 for other towns. Figure 22B Shows the predictive posterior fit of the Bayesian hierarchical model. AsFigure 22B As shown, the predictive posterior check shows that the mean of the graph-based regenerated samples of X aligns well with the observations other than {0,1}. This means that the fit is better for larger values of the random variable, yet the fit still produces values within the observed regions of 0 and 1.

[0236] C. Scenario Selection

[0237] The information gain related to AV performance is optimized across the ODD. However, the same technique can be applied to select a set of scenarios to maximize the information gain for a particular town.

[0238] Constraints:

[0239] The Bayesian hierarchical model allows the calculation of This which is then used to calculate the conditional entropy and information gain for any new scenario. As long as the scenario can be placed in the graph, the additional information provided by the scenario can be inferred.

[0240] Algorithm 1 details an example step for solving Equation 8. The conditional entropy is calculated up to the 90% confidence interval with an absolute error of 0.1. At each step, scenarios are added to the set by greedily looking at each scenario one by one, calculating the information obtained from showing the performance in that scenario compared to the previously selected scenarios, and adopting the scenario with the highest information gain for that step. This process is repeated until the information gain stops growing (within 90% confidence). This is why the algorithm is said to break ties arbitrarily.

[0241] Algorithm 1 - Greedy Algorithm:

[0242] Input: is a list of all possible scenarios and M is the inference model

[0243] Output: is an approximately optimal finite scenario selection

[0244] Initialization

[0245]

[0246] Compare the results of several runs implemented using Latin hypercube sampling and several randomly selected Algorithm 1. These results are presented in Figure 23 which shows a comparison of the information gain and stopping criteria between Algorithm 1, Latin hypercube sampling, and random. The vertical lines show the stopping criteria for Algorithm 1 and LHS. As Figure 23As shown, except for the first few scenarios, all the selections exhibit the property of diminishing returns, which confirms the submodular property of information gain. This may be due to the approximation when calculating this gain, which has a greater impact on values close to zero at the beginning. Additionally, using only a few observations may result in a degenerate version of the model, which can explain the lack of submodularity for the first few observations.

[0247] Regarding the validation of the choice of metric for evaluating the relevance of a set of scenarios, it is observed that information gain captures the fact that well-established methods such as LHS do perform better compared to random selection. Regardless of the ODD or activity being analyzed, this metric allows comparison of any scenario selection method. This metric also conveys a sense of progress in the evaluation activity and is intuitive and transparent (the evaluation can be stopped when no more significant information is received).

[0248] Figure 23 The main results in [reference] are as follows: In the case where all sampling methods maintain a stable level at the same information gain level, regardless of whether LHS requires 12 scenarios or approximately an additional 2% of the scenario space, the proposed algorithm uses only 9 scenarios or approximately 7% of the scenario space to obtain this amount, and the random method needs to explore at least another 2% of the space. This experiment was conducted on a small scale, but these differences can represent the gain for millions of scenarios in large-scale AV validation processing. The selected set contains scenarios that occur in 4 different towns but not all towns. Scenarios in towns 02 or 03 are not predicted to bring as much information as the scenarios in towns 01, 04, 05, and 06. When the AV is actually tested with the selected scenarios, the AV usually encounters 0 or 1 collision, but also involves 7 collisions in one scenario and 11 collisions in another scenario. Although the algorithm does not seek rare events, the scenarios at the tails of these distributions provide a large amount of information relevant to the system under test.

[0249] Experiment Summary

[0250] In summary, the scenario selection problem for AV performance evaluation can be reformulated as a submodular optimization problem on a graph. In this way, a statistical model can be used to represent AV performance, thus leveraging the progress defined by quantitative AV behavior. Additionally, a hierarchical Bayesian model can be fitted to the dataset generated by an open-source AV implementation, which indicates that this technique can be applied to any AV implementation.

[0251] Once the Bayesian hierarchical model is validated, the submodular property follows, and a set of scenarios can be constructed to utilize this property. Regardless of the ODD or activity being examined, information gain can be used to compare scenario selection methods.

[0252] Example Processing

[0253] Figure 24An example process 2400 for selecting test scenarios for evaluating the performance of an AV is shown. Process 2400 can be performed at least in part using the AV performance evaluation system 1304 (e.g., as described with reference to Figures 13-23 ).

[0254] According to process 2400, the computer system receives first data representing a plurality of scenarios for estimating the performance of a vehicle during autonomous operation (block 2402). As an example, each scenario can be a different corresponding combination of road conditions, obstacles, other vehicles, pedestrians, traffic flow, or environmental conditions. In some implementations, the autonomous operation can include autonomously navigating the vehicle from a first location to a second location.

[0255] The computer system determines for each scenario: (i) a first metric that indicates the observed performance of the vehicle in that scenario, where the first metric is determined based on at least one rule; and (ii) a second metric that indicates the degree of information gain associated with that scenario (block 2404).

[0256] In some implementations, the first metric can indicate information related to the performance of the AV in each scenario, such as the number of collisions of the AV, the number of traffic rule violations of the AV, the number and / or degree of route deviations of the AV, the number and / or degree of route blockages of the AV, and / or the time taken for the AV to cross the route.

[0257] In some implementations, the rule can include at least one traffic rule. Example traffic rules include rules for obeying signs or signals, driving with the traffic, and maintaining a legal speed.

[0258] In some implementations, the rule can include at least one safety rule. Example safety rules include rules for not colliding with other objects, maintaining a safe distance between other vehicles or pedestrians, and performing avoidance maneuvers (e.g., avoiding other vehicles or pedestrians) when necessary.

[0259] In some implementations, the rule can include at least one occupant comfort rule. Example occupant comfort rules include rules for accelerating or decelerating smoothly and turning smoothly.

[0260] In some implementations, the rule can include at least one vehicle performance rule. Example vehicle performance rules include rules for not exceeding specific speed, braking, and acceleration limits of the vehicle.

[0261] In some implementations, for each scenario, the degree of information gain associated with that scenario can correspond to a reduction in the entropy used for estimating the performance of the vehicle.

[0262] The computer system selects a subset of scenarios based on a first metric and a second metric (block 2406). In some implementations, the subset of scenarios can be used for AV testing or verification.

[0263] In some implementations, a subset of scenarios can be selected by identifying multiple candidate subsets of scenarios and determining a corresponding third metric for each candidate subset that indicates the degree of information gain associated with that candidate subset. Additionally, the candidate subset having the highest third metric among the candidate subsets can be selected as the subset of scenarios.

[0264] In some implementations, a subset of scenarios can be selected by constraining the number of scenarios in the subset of scenarios according to a budget metric. For example, the budget metric can specify the maximum number of scenarios that can be tested.

[0265] The computer system outputs second data indicating the subset of scenarios (block 2408).

[0266] In some implementations, according to process 2400, the computer system can determine the observed performance of at least one additional vehicle in each scenario of the subset of scenarios.

[0267] In some implementations, according to process 2400, the computer system can cluster multiple scenarios into multiple clusters. Additionally, a subset of scenarios is selected based on the clusters. In some implementations, the clusters can be based on principal component analysis.

[0268] In some implementations, according to process 2400, the computer system can determine the arrangement of scenarios based on the clusters. Additionally, a subset of scenarios can be selected based on the arrangement. In some implementations, the arrangement can be a Bayesian hierarchical model. In some implementations, the arrangement can be a Bayesian hierarchical model, a Bayesian network, a factor graph, and / or a hidden Markov model.

[0269] In some implementations, a subset of scenarios can be selected from the arrangement according to a greedy algorithm.

[0270] In the foregoing description, several embodiments have been described with reference to numerous specific details, which may vary according to implementation. Accordingly, the specification and drawings are to be regarded as illustrative rather than in a limiting sense. The sole and exclusive indication of the scope of the present invention, and what the applicant desires to be the scope of the present invention, is the literal and equivalent scope of the claims that issue from this application in the specific form of the issued claims, including any subsequent amendments. Any definition of terms expressly set forth herein for inclusion in such claims shall be construed to have the meaning such terms have as used in the claims. Additionally, when the term "further comprises" is used in the foregoing specification or the appended claims, the text following that phrase can be additional steps or entities, or sub-steps / sub-entities of the previously described steps or entities.

Claims

1. A method for a vehicle, comprising: Receiving, by a computer system, first data representing a plurality of scenarios for estimating the performance of a vehicle during autonomous operation; Using the computer system, for each scenario, determining: A first metric indicating the observed performance of the vehicle in that scenario, wherein the first metric is determined based on at least one rule, and A second metric indicating the degree of information gain associated with that scenario; Using the computer system, determining at least one of a Bayesian hierarchical model, a Bayesian network, a factor graph, and a hidden Markov model, wherein the plurality of scenarios are grouped according to a graphical representation including the at least one of the Bayesian hierarchical model, the Bayesian network, the factor graph, and the hidden Markov model; Using the computer system, selecting a subset of scenarios based on the first metric, the second metric, and the at least one of the Bayesian hierarchical model, the Bayesian network, the factor graph, and the hidden Markov model, wherein selecting the subset of scenarios includes determining that the subset of scenarios is associated with the maximum degree of information gain among the plurality of scenarios; and Using the computer system, outputting second data indicating the subset of scenarios.

2. The method according to claim 1, further comprising: Using the computer system, determining the observed performance of at least one additional vehicle in each scenario of the subset of scenarios.

3. The method according to claim 1, further comprising: Using the computer system, clustering the plurality of scenarios into a plurality of clusters, wherein the subset of scenarios is selected based on the clusters.

4. The method according to claim 3, further comprising: Using the computer system, determining the arrangement of the plurality of scenarios based on the clusters, wherein the subset of scenarios is selected based on the arrangement.

5. The method according to claim 4, wherein, The subset of scenarios is selected from the arrangement according to a greedy algorithm.

6. The method according to claim 1, wherein For each scenario, the degree of information gain associated with that scenario corresponds to a reduction in entropy used for estimating the performance of the vehicle.

7. The method according to claim 1, wherein Selecting the subset of scenarios includes: Identifying a plurality of candidate subsets of scenarios, For each candidate subset, determining a corresponding third metric indicating the degree of information gain associated with that candidate subset, and Selecting, from the candidate subsets, the candidate subset having the highest third metric among the third metrics as the subset of scenarios.

8. The method according to claim 1, wherein Selecting the subset of scenarios includes: Constraining the number of scenarios in the subset of scenarios according to a budget metric.

9. The method according to claim 1, wherein The at least one rule includes at least one of the following: At least one traffic rule, At least one safety rule, At least one occupant comfort rule, and At least one vehicle performance rule.

10. The method according to claim 1, wherein, The autonomous operation includes autonomously navigating the vehicle from a first location to a second location.

11. A system for a vehicle, comprising: At least one processor; And At least one non-transitory computer-readable medium storing instructions that, when executed by the at least one processor, cause the at least one processor to perform operations, the operations including: Receive first data representing a plurality of scenarios for estimating the performance of a vehicle during autonomous operation; For each scenario, determine: A first metric that indicates the observed performance of the vehicle in the scenario, where the first metric is determined based on at least one rule, and A second metric that indicates the degree of information gain associated with the scenario; Determine at least one of a Bayesian hierarchical model, a Bayesian network, a factor graph, and a hidden Markov model, where the plurality of scenarios are grouped according to a graphical representation including the at least one of the Bayesian hierarchical model, the Bayesian network, the factor graph, and the hidden Markov model; Select a subset of scenarios based on the first metric, the second metric, and the at least one of the Bayesian hierarchical model, the Bayesian network, the factor graph, and the hidden Markov model, where selecting the subset of scenarios includes determining that the subset of scenarios is associated with the maximum degree of information gain among the plurality of scenarios; and Output second data indicating the subset of scenarios.

12. At least one non-transitory computer-readable medium storing instructions that, when executed by at least one processor, cause the at least one processor to perform operations including: Receive first data representing a plurality of scenarios for estimating the performance of a vehicle during autonomous operation; For each scenario, determine: A first metric that indicates the observed performance of the vehicle in the scenario, where the first metric is determined based on at least one rule, and A second metric that indicates the degree of information gain associated with the scenario; Determine at least one of a Bayesian hierarchical model, a Bayesian network, a factor graph, and a hidden Markov model, where the plurality of scenarios are grouped according to a graphical representation including the at least one of the Bayesian hierarchical model, the Bayesian network, the factor graph, and the hidden Markov model; Select a subset of scenarios based on the first metric, the second metric, and the at least one of the Bayesian hierarchical model, the Bayesian network, the factor graph, and the hidden Markov model, where selecting the subset of scenarios includes determining that the subset of scenarios is associated with the maximum degree of information gain among the plurality of scenarios; and Output second data indicating the subset of scenarios.

13. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 10.

Images