Autonomous vehicle simulation to improve safety and reliability of autonomous vehicles

Abstract

A system is described that may include a first database, a simulator, and a second database. The first database may store data indicating the operation of at least one module within a computing device of an autonomous vehicle. The simulator may receive the stored data from the first database. The simulator may generate a simulation of the operation of the at least one module based on the received data. The simulator may identify at least a portion of the simulation that indicates a deviation between the collected data and the simulated operation of the autonomous vehicle. The simulator may analyze the at least a portion of the simulation to generate metrics for the at least a portion of the simulation. The metrics may be used to avoid another deviation between the collected data and the simulated operation of the autonomous vehicle. The second database may store the metrics.

Description

Technical Field

[0001] The present description generally relates to generating metrics during simulation of operation of one or more modules of an autonomous vehicle and using these metrics to improve the safety and reliability of the autonomous vehicle. Background Art

[0002] There are various simulation techniques for testing autonomous vehicles (AVs). In conventional techniques, a simulation of a test environment is created to enable the AV to detect objects in the environment and navigate in different weather conditions (e.g., rainy or daytime). However, such techniques are not sufficient to fully improve the safety and reliability of AVs. For example, such conventional techniques often fail to diagnose and address the following two situations: (a) erroneously generating an alert when a collision of the AV is unlikely (i.e., a false positive); and (b) failing to generate an alert when a collision is likely to occur (i.e., a missed negative). In addition, when a takeover occurs in an attempt to prevent a collision, such techniques fail to assess whether a collision would have been likely if the takeover had not occurred, and thus may often generate inaccurate alerts. Summary of the Invention

[0003] The present invention provides a method comprising: receiving, using one or more processors, data collected by a computing device of an autonomous vehicle during operation of the autonomous vehicle; generating, using the one or more processors and based on the received data, a simulation of the operation of the autonomous vehicle; identifying, using the one or more processors, at least a portion of the simulation that indicates a deviation between the collected data and the simulated operation of the autonomous vehicle; analyzing, using the one or more processors, at least a portion of the simulation to generate a metric for the at least a portion of the simulation; and sending, using the one or more processors, the metric to the computing device, wherein the computing device is configured to use the metric to avoid another deviation between the collected data and the simulated operation of the autonomous vehicle.

[0004] The present invention provides a system comprising: a first database for storing data indicating the operation of at least one module within a computing device of an autonomous vehicle; a simulator comprising one or more processors, wherein the one or more processors are configured to: receive the stored data from the first database, generate a simulation of the operation of the at least one module based on the received data, identify at least a portion of the simulation, the at least a portion indicating a deviation between the collected data and the simulated operation of the autonomous vehicle, and analyze the at least a portion of the simulation to generate a metric for the at least a portion of the simulation, wherein the metric can be used to avoid another deviation between the collected data and the simulated operation of the autonomous vehicle; and a second database configured to store the metric.

[0005] The present invention provides a non-transitory computer-readable medium storing instructions, which, when executed by at least one programmable processor, causes the at least one programmable processor to perform operations including: receiving data collected by a computing device of an autonomous vehicle during operation of the autonomous vehicle; generating a simulation of the operation of the autonomous vehicle based on the received data; identifying at least a portion of the simulation that indicates a deviation between the collected data and the simulated operation of the autonomous vehicle; analyzing at least a portion of the simulation to generate a metric for at least a portion of the simulation; and sending the metric to the computing device, wherein the computing device is configured to use the metric to avoid another deviation between the collected data and the simulated operation of the autonomous vehicle.

[0006] The present invention provides a method comprising: receiving, using one or more processors, data collected by a computing device of an autonomous vehicle during operation of the autonomous vehicle; generating, using the one or more processors and based on the received data, a simulation of the operation of the autonomous vehicle; identifying, using the one or more processors, at least a portion of the simulation that indicates a takeover from an autonomous driving mode in response to the computing device generating an alert indicating a potential collision; analyzing, using the one or more processors, at least a portion of the simulation to determine a metric indicating whether a collision would have occurred if the takeover had not occurred; and sending, using the one or more processors, the metric to the computing device, wherein the computing device is configured to use the metric to improve the accuracy of future alerts indicating another potential collision.

[0007] The present invention provides a system comprising: a first database configured to store data received from a computing device of an autonomous vehicle, wherein the stored data indicates the operation of the autonomous vehicle; a simulator comprising one or more processors, wherein the one or more processors are configured to: receive the stored data from the first database, generate a simulation of the operation of the autonomous vehicle based on the received data, identify at least a portion of the simulation, which indicates a takeover from an autonomous driving mode in response to the computing device generating an alert indicating a potential collision, and analyze the at least a portion of the simulation to determine a metric indicating whether a collision would have occurred if the takeover had not occurred; and a second database configured to store the metric.

[0008] The present invention provides a non-transitory computer-readable medium having instructions stored thereon, which, when executed by at least one programmable processor, causes the at least one programmable processor to perform operations including: receiving data collected by a computing device of an autonomous vehicle during operation of the autonomous vehicle; generating a simulation of the operation of the autonomous vehicle based on the received data; identifying at least a portion of the simulation that indicates a takeover from an autonomous driving mode in response to the computing device generating an alert indicating a potential collision; analyzing at least a portion of the simulation to determine a metric indicating whether a collision would have occurred if the takeover had not occurred; and sending the metric to the computing device, wherein the computing device is configured to use the metric to improve the accuracy of future alerts indicating another potential collision. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 An example of an autonomous vehicle (AV) with autonomous capabilities is shown.

[0010] Figure 2 An example "cloud" computing environment is illustrated.

[0011] Figure 3 An example computer system.

[0012] Figure 4 An example architecture of an AV is shown.

[0013] Figure 5 Shows examples of inputs and outputs that a perception module can use.

[0014] Figure 6 An example of a LiDAR system is shown.

[0015] Figure 7 The LiDAR system is shown in operation.

[0016] Figure 8 Additional details of the operation of the LiDAR system are shown.

[0017] Figure 9 A block diagram showing the relationship between the inputs and outputs of the planning module.

[0018] Figure 10 Shows the directed graph used in path planning.

[0019] Figure 11 A block diagram showing the inputs and outputs of the control module.

[0020] Figure 12 A block diagram showing the inputs, outputs, and components of the controller.

[0021] Figure 13 A computing environment is shown for generating simulation data for generating metrics used by a computing device of an AV to make the AV safe and reliable.

[0022] Figure 14 A process is shown for making an AV safe and reliable by reducing false positives and false negatives generated by the AV's computing device.

[0023] Figure 15 A specific example of avoiding false alarms generated by a computing device of an AV is shown.

[0024] Figure 16 A specific example of avoiding false negatives generated by a computing device of an AV is shown.

[0025] Figure 17 A process is shown for generating metrics to improve the accuracy of future alerts indicating another potential collision.

[0026] Figure 18 An example of a computational landscape for generating metrics to improve the accuracy of future alerts indicating another potential collision is shown.

[0027] The same reference numerals in the various drawings indicate the same elements or components. DETAILED DESCRIPTION

[0028] In the following description, for purposes of explanation, numerous specific details are set forth to provide a thorough understanding. However, it will be apparent that some implementations may be practiced without these specific details. In other instances, well-known configurations and devices are shown in block diagram form to avoid unnecessarily obscuring the various implementations.

[0029] In the accompanying drawings, for ease of description, the specific arrangement or order of schematic elements, such as those representing devices, modules, instruction blocks, and data elements, is shown. However, it will be understood by those skilled in the art that the specific ordering or arrangement of schematic elements in the accompanying drawings does not necessarily require a specific processing order or sequence, or separation of processing processes. Furthermore, the inclusion of schematic elements in the accompanying drawings does not mean that such elements are required in all embodiments, nor does it mean that the features represented by such elements cannot be included in some embodiments or cannot be combined with other elements in some embodiments.

[0030] In addition, in the accompanying drawings, connecting elements, such as solid or dotted lines or arrows, are used to illustrate the connection, relationship or association between two or more other schematic elements, and the absence of any such connecting elements does not mean that there can be no connection, relationship or association. In other words, the connection, relationship or association between some elements are not shown in the accompanying drawings so as not to obscure the present invention. In addition, for ease of explanation, a single connecting element is used to represent multiple connections, relationships or associations between elements. For example, if a connecting element represents the communication of a signal, data or instruction, it will be understood by those skilled in the art that the element represents one or more signal paths (e.g., buses) that may be needed to affect the communication.

[0031] Reference will now be made in detail to the embodiments, examples of which are illustrated in the accompanying drawings. In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the various embodiments described. However, it will be apparent to one of ordinary skill in the art that the various embodiments described may be practiced without these specific details. In other instances, well-known methods, procedures, components, circuits, and networks have not been described in detail so as not to unnecessarily obscure aspects of the embodiments.

[0032] Several of the features described below can be used independently of each other or in combination with any other features. However, any individual feature may not address all of the above problems, or may address only one of the above problems. Some of the problems discussed above may not be fully addressed by any one of the features described herein. Although headings are provided, information related to a heading but not found in that heading may be found elsewhere in this description. This document describes embodiments according to the following summary:

[0033] 1. General Overview

[0034] 2. System Overview

[0035] 3. Autonomous Vehicle Architecture

[0036] 4. Autonomous Vehicle Input

[0037] 5. Autonomous Vehicle Planning

[0038] 6. Autonomous Vehicle Control

[0039] 7. A simulation environment that improves the safety and reliability of AVs by reducing the generation of false positives and false negatives

[0040] 8. A simulation environment that improves the safety and reliability of AVs by increasing the accuracy of future alerts indicating another potential collision

[0041] General Overview

[0042] In one aspect, data collected by a computing device of an autonomous vehicle (AV) is used to generate a simulation of the AV. The simulation is compared to the collected data to determine a deviation between the collected data and the simulation. In one example, such a deviation may represent a false positive by the computing device, wherein the collected data indicates that the AV detected a particular type of event or scenario (e.g., a potential collision) (e.g., generated an alert for the particular type of event or scenario), while the simulated operation indicates that the detection (e.g., alert) is false. In another example, such a deviation may represent a false negative by the computing device, wherein the collected data indicates that the computing device failed to detect a particular type of event or scenario (e.g., a potential collision) (e.g., failed to indicate an alert for the particular type of event or scenario), while the simulated operation indicates the event or scenario (e.g., potential collision). Identifying such false positives and false negatives may be used to generate a metric that prevents the computing device from generating such false positives or false negatives.

[0043] Although the detection of an event or scenario is described as being communicated to the driver via an alert, in other implementations, the detection may not be communicated to the driver. Detection of a collision may be performed by a collision avoidance module within the AV's software stack. Although the event or scenario is described as a collision, in other implementations, the event or scenario may be any other event or scenario detected by any other module within the software stack, such as a localization module, a planning module, a perception module, and / or a control module.

[0044] This aspect provides various advantages. For example, preventing false positives can reduce (e.g., minimize) the number of false detections (e.g., alerts) of impending events or scenarios (e.g., collisions) generated by the AV's computing device. Reducing the number of false detections (e.g., alerts) in this way can reduce the number of unnecessary interventions (e.g., by the driver or remote controller) required during the operation of the AV. This improves the reliability of the AV and instills confidence in the operation of the AV. Furthermore, preventing false negatives can ensure that the AV's computing device detects a specific event or scenario (e.g., a potential collision) (e.g., generates an alert indicating the specific event or scenario). This can advantageously prevent undesirable events or scenarios (e.g., prevent a collision). Preventing undesirable events or scenarios (e.g., collisions) in this way can enhance the safety and reliability of the AV.

[0045] In another aspect, a simulation of the AV is generated using data collected by a computing device of the AV. When a takeover from an autonomous driving mode occurs in response to detecting an event or scenario (e.g., a potential collision) (e.g., generating an alert indicating the event or scenario using the computing device), the simulation is examined to identify a portion of the simulation. The portion of the simulation is analyzed to determine a metric that indicates whether the event or scenario (e.g., a collision) would have occurred if the takeover had not occurred (i.e., the probability of such an event or scenario, such as a collision, occurring is greater than a threshold). Such metrics are sent to a computing device that can use the metrics to improve the accuracy of future detections of another event or scenario (e.g., another potential collision) (e.g., an alert indicating another event or scenario).

[0046] Although the detection of an event or scenario is described as being communicated to the driver via an alert, in other implementations, the detection may not be communicated to the driver. Detection of a collision may be performed by a collision avoidance module within the AV's software stack. Although the event or scenario is described as a collision, in other implementations, the event or scenario may be any other event or scenario detected by any other module within the software stack, such as a localization module, a planning module, a perception module, and / or a control module.

[0047] This aspect also provides various advantages. For example, the accuracy of future detection of another event or scenario (e.g., a potential collision) (e.g., an alert indicating another event or scenario) is improved, which in turn reduces the event or scenario (e.g., collision) experienced by the AV, thereby making the AV safer and more reliable. In addition, in some implementations, such detection (e.g., generation of an alert) can initiate automatic takeover of the remote system, which is advantageous in situations where there is no human driver or where the human driver is negligent or careless. This further enhances the reliability of the AV. In some examples, such detection (e.g., generation of an alert) can be performed in response to a change in the driver's vital signs (e.g., where the driver is experiencing a medical difficulty such as a heart attack), in which case the alert can initiate automatic takeover of the remote system.

[0048] The above aspects are further advantageous. For example, in some implementations, the data collected can be operational data for any AV manufactured by any manufacturer (not necessarily the manufacturer that generates the metrics to improve the reliability and safety of the AV). Thus, logs of operational data for AVs manufactured by any manufacturer can be collected to present improvements to the AV, thereby enabling the technical implementations described herein to benefit the entire AV industry. In addition, the simulation system can be modified to simulate any module within the AV's software stack, and is therefore not limited to the simulation of any specific one or more modules. This advantageously allows any module to be inserted into the simulation system without modifying (or without significantly modifying) the architecture of the simulation system. The lack of need to modify the architecture prevents users of the simulation system from rewriting the code used for the simulation, thereby making the simulation system easier to use. In addition, the technology described herein can be further extended from AVs to any other robotic systems to improve the reliability and safety of such robotic systems.

[0049] System Overview

[0050] Figure 1 An example of an autonomous vehicle (AV) 100 having autonomous capabilities is shown.

[0051] As used herein, the term "autonomous capability" refers to a function, feature, or facility that enables a vehicle to operate partially or fully without real-time human intervention, including but not limited to fully autonomous vehicles, highly autonomous vehicles, partially autonomous vehicles, and conditionally autonomous vehicles.

[0052] As used herein, an AV is a vehicle with autonomous capabilities.

[0053] As used herein, "vehicle" includes any mode of transport for goods or people, such as a car, bus, train, airplane, drone, truck, boat, ship, submersible, or spacecraft. An unmanned car is an example of a vehicle.

[0054] As used herein, a "trajectory" refers to a path or route that an AV takes from a first spatiotemporal location to a second spatiotemporal location. In embodiments, the first spatiotemporal location is referred to as an initial location or starting location, and the second spatiotemporal location is referred to as a destination, final location, target, target position, or target location. In some examples, a trajectory is comprised of one or more road segments (e.g., sections of a road), and each road segment is comprised of one or more blocks (e.g., a lane or portion of an intersection). In embodiments, a spatiotemporal location corresponds to a real-world location. For example, a spatiotemporal location is a pickup or drop-off location for picking up or dropping off people or cargo.

[0055] As used herein, "sensor(s)" includes one or more hardware components for detecting information related to the sensor's surroundings. Some hardware components may include sensing components (e.g., image sensors, biometric sensors), transmitting and / or receiving components (e.g., laser or radio frequency wave transmitters and receivers), electronic components (e.g., analog-to-digital converters), data storage devices (e.g., RAM and / or non-volatile memory), software or firmware components and data processing components (e.g., application specific integrated circuits), microprocessors, and / or microcontrollers.

[0056] As used herein, a “scene description” is a data structure (e.g., a list) or data stream that includes one or more classified or labeled objects detected by one or more sensors on an AV vehicle, or one or more classified or labeled objects provided by a source external to the AV.

[0057] As used herein, a "road" is a physical area that can be traversed by a vehicle and can correspond to a named thoroughfare (e.g., a city street, an interstate highway, etc.) or can correspond to an unnamed thoroughfare (e.g., a driveway within a house or office building, a section of a parking lot, a section of a vacant parking lot, a dirt road in a rural area, etc.). Because some vehicles (e.g., four-wheel drive pickup trucks, off-road vehicles (SUVs), etc.) can traverse a variety of physical areas that are not particularly suitable for vehicle travel, a "road" can be any physical area that has not been formally defined as a thoroughfare by a municipality or other governmental or administrative agency.

[0058] As used herein, a “lane” is a portion of a road that can be traversed by a vehicle. Lanes are sometimes identified based on lane markings. For example, a lane may correspond to most or all of the space between lane markings, or only a portion of the space between lane markings (e.g., less than 50%). For example, a road with lane markings that are far apart may accommodate two or more vehicles, allowing one vehicle to pass another without crossing the lane markings, and thus may be interpreted as a lane narrower than the space between lane markings, or as having two lanes between lanes. Lanes may also be interpreted in the absence of lane markings. For example, lanes may be defined based on physical features of the environment (e.g., rocks and trees along an avenue in a rural area, or natural obstacles to be avoided, for example, in less developed areas). Lanes may also be interpreted independently of lane markings or physical features. For example, a lane may be interpreted based on an arbitrary path through an area without obstacles that would otherwise lack features that would be interpreted as lane boundaries. In an example scenario, an AV may interpret a lane as passing through an obstacle-free portion of a field or open space. In another example scenario, an AV may interpret a lane through a wide (e.g., wide enough for two or more lanes) road that does not have lane markings. In this scenario, the AV may communicate lane-related information to other AVs so that the other AVs can use the same lane information to coordinate path planning between the AVs.

[0059] The term “Over-the-Air (OTA) Client” includes any AV, or any electronic device (e.g., a computer, controller, IoT device, Electronic Control Unit (ECU)) embedded in, coupled to, or communicating with an AV.

[0060] The term "Over-the-Air (OTA) Update" means any update, change, deletion, or addition to software, firmware, data, or configuration settings, or any combination thereof, delivered to an OTA Client using proprietary and / or standardized wireless communication technologies including, but not limited to, cellular mobile communications (e.g., 2G, 3G, 4G, 5G), radio wireless local area networks (e.g., WiFi), and / or satellite Internet.

[0061] The term “edge node” refers to one or more edge devices coupled to a network that provide a portal for communicating with AVs and can communicate with other edge nodes and cloud-based computing platforms to schedule and deliver OTA updates to OTA clients.

[0062] The term "edge device" refers to a device that implements an edge node and provides a physical wireless access point (AP) to the core network of an enterprise or service provider (e.g., Verizon, AT&T). Examples of edge devices include, but are not limited to, computers, controllers, transmitters, routers, routing switches, integrated access devices (IADs), multiplexers, metropolitan area network (MAN), and wide area network (WAN) access devices.

[0063] “One or more” includes a function performed by one element, a function performed by multiple elements, such as in a distributed manner, several functions performed by one element, several functions performed by several elements, or any combination of the foregoing.

[0064] It will also be understood that although in some cases the terms "first," "second," etc. are used to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first contact may be referred to as a second contact, and similarly, a second contact may be referred to as a first contact without departing from the scope of the various described embodiments. The first contact and the second contact are both contacts, but they are not the same contact.

[0065] The terms used in the description of the various embodiments described herein are for describing specific embodiments only and are not intended to be limiting. As used in the description of the various embodiments described and in the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" as used herein refers to and includes any and all possible combinations of one or more of the associated list items. It will also be understood that the terms "comprises," "comprising," "having," and / or "having" as used in this description specify the presence of the stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0066] As used herein, the term "if" may alternatively be understood as meaning in that case, at that time, or in response to being detected, or in response to being determined, as the context requires. Similarly, the phrase "if it has been determined" or "if [the condition or event] has been detected" may be understood as meaning "upon determination" or "in response to being determined" or "upon detection of [the condition or event]" or "in response to detecting [the condition or event]," as the context requires.

[0067] As used herein, an AV system refers to an AV and the hardware, software, stored data, and data generated in real time to support AV operations. In an embodiment, the AV system is incorporated into the AV. In an embodiment, the AV system is distributed across multiple locations. For example, some software of the AV system is combined in a similar manner to the following: Figure 3 The described cloud computing environment 300 is implemented in a cloud computing environment.

[0068] In general, this document describes technologies applicable to any vehicle with one or more autonomous capabilities, including fully autonomous vehicles, highly autonomous vehicles, and conditionally autonomous vehicles, such as so-called Level 5, Level 4, and Level 3 vehicles (see SAE International Standard J3016: Classification and Definitions of Terms Relating to Automated Driving Systems for Road-Based Motor Vehicles, which is incorporated herein by reference in its entirety for more details on vehicle autonomy levels). The technologies described in this document are also applicable to partially autonomous vehicles and driver-assisted vehicles, such as so-called Level 2 and Level 1 vehicles (see SAE International Standard J3016: Classification and Definitions of Terms Relating to Automated Driving Systems for Road-Based Motor Vehicles). In embodiments, one or more Level 1, Level 2, Level 3, Level 4, and Level 5 vehicle systems may automatically perform certain vehicle operations (e.g., steering, braking, and using maps) under certain operating conditions based on processing of sensor inputs. The technologies described in this document can benefit vehicles at all levels, from fully autonomous vehicles to human-operated vehicles.

[0069] AVs have advantages over vehicles that require human drivers. One advantage is safety. For example, in 2016, the United States experienced 6 million automobile accidents, 2.4 million injuries, 40,000 deaths, and 13 million vehicle collisions, with an estimated societal cost of more than $910 billion. From 1965 to 2015, the number of U.S. traffic fatalities decreased from approximately 6 to approximately 1 per 100 million miles traveled, partly due to additional safety measures deployed in vehicles. For example, an extra half-second of warning about an impending collision is believed to mitigate 60% of front-to-rear collisions. However, passive safety features (e.g., seatbelts, airbags) may have reached their limits in improving these numbers. Therefore, active safety measures such as automated control of vehicles are a possible next step in improving these statistics. Because human drivers are believed to be the cause of serious pre-crash events in 95% of collisions, automated driving systems have the potential to achieve better safety outcomes by, for example: reliably identifying and avoiding emergency situations better than humans; making better decisions than humans, obeying traffic laws better than humans, and predicting future events better than humans; and reliably controlling a vehicle better than humans.

[0070] refer to Figure 1 , the AV system 120 causes the AV 100 to operate along a trajectory 198 through an environment 190 to a destination 199 (sometimes referred to as a final location) while avoiding objects (e.g., natural obstacles 191, vehicles 193, pedestrians 192, cyclists, and other obstacles) and obeying the rules of the road (e.g., operating rules or driving preferences).

[0071] In an embodiment, the AV system 120 includes a device 101 for receiving and operating operational commands from a computing processor 146. The term "operating command" is used to refer to an executable instruction (or set of instructions) that causes a vehicle to perform an action (e.g., a driving maneuver). The operating command may include, but is not limited to, instructions for causing the vehicle to start moving forward, stop moving forward, start moving backward, stop moving backward, accelerate, decelerate, make a left turn, and make a right turn. In an embodiment, the computing processor 146 is connected to the vehicle 146 with reference to the following. Figure 3 The processor 304 is similarly described. Examples of devices 101 include steering controls 102, brakes 103, gears, an accelerator pedal or other acceleration control mechanism, windshield wipers, side door locks, window controls, and turn indicators.

[0072] In an embodiment, the AV system 120 includes sensors 121 for measuring or inferring attributes of the state or condition of the AV 100, such as the AV's position, linear and angular velocity and acceleration, and heading (e.g., the direction of the front end of the AV 100). Examples of sensors 121 are GPS, an inertial measurement unit (IMU) for measuring vehicle linear acceleration and angular rate, wheel rate sensors for measuring or estimating wheel slip, wheel brake pressure or brake torque sensors, engine torque or wheel torque sensors, and steering angle and angular rate sensors.

[0073] In an embodiment, the sensors 121 also include sensors for sensing or measuring properties of the AV's environment, such as a monocular or stereo camera 122 in the visible, infrared, or thermal (or both) spectrum, a LiDAR 123, a RADAR, an ultrasonic sensor, a time-of-flight (TOF) depth sensor, a velocity sensor, a temperature sensor, a humidity sensor, and a precipitation sensor.

[0074] In an embodiment, the AV system 120 includes a data storage unit 142 and a memory 144 for storing machine instructions related to the computing processor 146 or data collected by the sensor 121. In an embodiment, the data storage unit 142 is combined with the following Figure 3ROM 308 or storage device 310 described above. In an embodiment, memory 144 is similar to main memory 306 described below. In an embodiment, data storage unit 142 and memory 144 store historical, real-time, and / or predictive information about environment 190. In an embodiment, the stored information includes maps, driving performance, traffic congestion updates, or weather conditions. In an embodiment, data related to environment 190 is transmitted to AV 100 via a communication channel from remote database 134.

[0075] In an embodiment, the AV system 120 includes communication devices 140 for transmitting measured or inferred attributes of the state and condition of other vehicles (such as position, linear and angular velocity, linear and angular acceleration, and linear and angular heading) to the AV 100. These devices include vehicle-to-vehicle (V2V) and vehicle-to-infrastructure (V2I) communication devices, as well as devices for wireless communication via point-to-point or ad hoc networks, or both. In an embodiment, the communication devices 140 communicate across the electromagnetic spectrum (including radio and optical communications) or other media (e.g., air and acoustic media). The combination of vehicle-to-vehicle (V2V) and vehicle-to-infrastructure (V2I) communications (and in some embodiments, one or more other types of communications) is sometimes referred to as vehicle-to-everything (V2X) communication. V2X communications typically conform to one or more communication standards for communication with and between AVs.

[0076] In an embodiment, the communication device 140 includes a communication interface. For example, a wired, wireless, WiMAX, Wi-Fi, Bluetooth, satellite, cellular, optical, near-field, infrared, or radio interface. The communication interface transmits data from the remote database 134 to the AV system 120. In an embodiment, the remote database 134 is embedded in the cloud computing environment 200, such as Figure 2 The communication interface 140 transmits data collected from the sensors 121 or other data related to the operation of the AV 100 to the remote database 134. In an embodiment, the communication interface 140 transmits information related to remote operation to the AV 100. In some embodiments, the AV 100 communicates with other remote (e.g., "cloud") servers 136.

[0077] In an embodiment, the remote database 134 also stores and transmits digital data (e.g., data storing roads and street locations, etc.) This data is stored in the memory 144 on the AV 100 or transmitted from the remote database 134 to the AV 100 via a communication channel.

[0078] In an embodiment, the remote database 134 stores and transmits historical information regarding driving attributes (e.g., speed and acceleration rate profiles) of vehicles that have previously traveled along the trajectory 198 at similar times of day. In one implementation, such data may be stored in the memory 144 on the AV 100 or transmitted from the remote database 134 to the AV 100 via a communication channel.

[0079] The computing processor 146 located onboard the AV 100 algorithmically generates control actions based on real-time sensor data and a priori information, enabling the AV system 120 to perform its autonomous driving capabilities.

[0080] In an embodiment, the AV system 120 includes a computer peripheral device 132 connected to the computing processor 146 for providing information and reminders to a user of the AV 100 (e.g., an occupant or a remote user) and receiving input from the user. In an embodiment, the peripheral device 132 is similar to the one described below with reference to Figure 3 The display 312, input device 314 and cursor control 316 are discussed. The connection can be wireless or wired. Any two or more interface devices can be integrated into a single device.

[0081] In an embodiment, the AV system 120 receives and enforces the occupant's privacy level, for example, as specified by the occupant or stored in a profile associated with the occupant. The occupant's privacy level determines how the use of specific information associated with the occupant (e.g., occupant comfort data, biometric data, etc.) stored in the occupant's profile and / or stored on the cloud server 136 and associated with the occupant's profile is permitted. In an embodiment, the privacy level specifies specific information associated with the occupant that is deleted once the ride is complete. In an embodiment, the privacy level specifies specific information associated with the occupant and identifies one or more entities that are authorized to access the information. Examples of specified entities that are authorized to access the information may include other AVs, third-party AV systems, or any entity that can potentially access the information.

[0082] The occupant's privacy level can be specified at one or more levels of granularity. In one embodiment, the privacy level identifies specific information to be stored or shared. In one embodiment, the privacy level applies to all information associated with the occupant, allowing the occupant to specify that her personal information not be stored or shared. The designation of entities permitted to access specific information can also be specified at various levels of granularity. The various sets of entities permitted to access specific information can include, for example, other AVs, cloud servers 136, specific third-party AV systems, and the like.

[0083] In an embodiment, the AV system 120 or cloud server 136 determines whether the AV 100 or another entity can access certain information associated with an occupant. For example, a third-party AV system seeking to access occupant input related to a particular spatiotemporal location must obtain authorization, for example from the AV system 120 or cloud server 136, to access information associated with the occupant. For example, the AV system 120 uses the occupant's designated privacy level to determine whether occupant input related to the spatiotemporal location can be presented to a third-party AV system, the AV 100, or another AV. This enables the occupant's privacy level to specify which other entities are permitted to receive data related to the occupant's actions or other data associated with the occupant.

[0084] Figure 2 Illustrate an example "cloud" computing environment. Cloud computing is a service delivery model that provides convenient, on-demand access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) over a network. In a typical cloud computing system, one or more large cloud data centers house the machines used to deliver the services provided by the cloud. Now refer to Figure 2 , cloud computing environment 200 includes cloud data centers 204a, 204b, and 204c interconnected by a cloud 202. Cloud data centers 204a, 204b, and 204c provide cloud computing services to computer systems 206a, 206b, 206c, 206d, 206e, and 206f connected to cloud 202.

[0085] The cloud computing environment 200 includes one or more cloud data centers. Typically, a cloud data center (e.g. Figure 2 The cloud data center 204a shown in FIG refers to a cloud (eg Figure 2 The physical arrangement of servers in a cloud 202 (or a specific portion of a cloud) as shown in FIG. For example, servers are physically arranged into rooms, groups, rows, and racks in a cloud data center. A cloud data center has one or more zones, which include one or more server rooms. Each room has one or more rows of servers, and each row includes one or more racks. Each rack includes one or more individual server nodes. In some implementations, servers in zones, rooms, racks, and / or rows are divided into groups based on the physical infrastructure requirements of the data center facility, including power, energy, heat, heat sources, and / or other requirements. In an embodiment, the server nodes are similar to Figure 3 The cloud data center 204a has many computing systems distributed across multiple racks.

[0086] Cloud 202 includes cloud data centers 204a, 204b, and 204c, as well as networks and network resources (e.g., network devices, nodes, routers, switches, and network cables) used to connect cloud data centers 204a, 204b, and 204c and facilitate access to cloud computing services by computing systems 206a-f. In embodiments, the network represents any combination of one or more local networks, wide area networks, or internetworks connected by wired or wireless links deployed using terrestrial or satellite connections. Data exchanged over the network is transmitted using a variety of network layer protocols, such as Internet Protocol (IP), Multiprotocol Label Switching (MPLS), Asynchronous Transfer Mode (ATM), Frame Relay, etc. Furthermore, in embodiments where the network represents a combination of multiple subnetworks, different network layer protocols are used on each underlying subnetwork. In some embodiments, the network represents one or more interconnected internetworks (e.g., the public Internet, etc.).

[0087] Computing systems 206a-f, or cloud computing service consumers, are connected to the cloud 202 via network links and network adapters. In embodiments, computing systems 206a-f are implemented as various computing devices, such as servers, desktops, laptops, tablets, smartphones, Internet of Things (IoT) devices, autonomous vehicles (including cars, drones, space shuttles, trains, buses, etc.), and consumer electronics. In embodiments, computing systems 206a-f are implemented in other systems or as part of other systems.

[0088] Figure 3 Illustrated is a computer system 300. In implementation, the computer system 300 is a special-purpose computing device. The special-purpose computing device is hard-wired to perform these techniques, or includes a digital electronic device such as one or more application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs) that is permanently programmed to perform the above-mentioned techniques, or may include one or more general-purpose hardware processors that are programmed to perform these techniques according to program instructions in firmware, memory, other memory, or a combination thereof. Such a special-purpose computing device may also combine customized hard-wired logic, ASICs, or FPGAs with customized programming to perform these techniques. In various embodiments, the special-purpose computing device is a desktop computer system, a portable computer system, a handheld device, a network device, or any other device that includes hard-wired and / or program logic to implement these techniques.

[0089] In an embodiment, computer system 300 includes a bus 302 or other communication mechanism for communicating information, and a processor 304 coupled to bus 302 for processing information. Processor 304 is, for example, a general-purpose microprocessor. Computer system 300 also includes a main memory 306, such as a random access memory (RAM) or other dynamic storage device, coupled to bus 302 to store information and instructions, which are executed by processor 304. In one implementation, main memory 306 is used to store temporary variables or other intermediate information during the execution of instructions to be executed by processor 304. When these instructions are stored in a non-transitory storage medium accessible to processor 304, computer system 300 becomes a special-purpose machine customized to perform the operations specified in the instructions.

[0090] In an embodiment, computer system 300 also includes a read-only memory (ROM) 308 or other static storage device connected to bus 302 for storing static information and instructions for processor 304. A storage device 310, such as a magnetic disk, optical disk, solid-state drive, or three-dimensional cross-point memory, is provided and connected to bus 302 for storing information and instructions.

[0091] In an embodiment, the computer system 300 is coupled via a bus 302 to a display 312, such as a cathode ray tube (CRT), a liquid crystal display (LCD), a plasma display, a light emitting diode (LED) display, or an organic light emitting diode (OLED) display for displaying information to a computer user. An input device 314, including alphanumeric and other keys, is coupled to the bus 302 for communicating information and command selections to the processor 304. Another type of user input device is a cursor controller 316, such as a mouse, a trackball, a touch-sensitive display, or cursor direction keys, for communicating direction information and command selections to the processor 304 and for controlling movement of a cursor on the display 312. Such input devices typically have two degrees of freedom along two axes, a first axis (e.g., an x-axis) and a second axis (e.g., a y-axis), which allow the device to specify a position on a plane.

[0092] According to one embodiment, the techniques herein are performed by computer system 300 in response to processor 304 executing one or more sequences of one or more instructions contained in main memory 306. These instructions are read into main memory 306 from another storage medium, such as storage device 310. Execution of the sequences of instructions contained in main memory 306 causes processor 304 to perform the process steps described herein. In alternative embodiments, hard-wired circuitry is used in place of or in combination with software instructions.

[0093] As used herein, the term "storage medium" refers to any non-transitory medium that stores data and / or instructions that cause a machine to operate in a specific manner. Such storage media include non-volatile media and / or volatile media. Non-volatile media include, for example, optical disks, magnetic disks, solid-state drives, or three-dimensional cross-point memories, such as storage device 310. Volatile media include dynamic memories, such as main memory 306. Common forms of storage media include, for example, floppy disks, disks, hard disks, solid-state drives, magnetic tape or any other magnetic data storage medium, CD-ROMs, any other optical data storage medium, any physical medium with a hole pattern, RAM, PROM and EPROM, FLASH-EPROM, NV-RAM, or any other memory chip or storage cartridge.

[0094] Storage media are distinct from transmission media, but can be used in conjunction with them. Transmission media participate in the transmission of information between storage media. Examples of transmission media include coaxial cables, copper wires, and optical fibers, including the wires that comprise bus 302. Transmission media can also take the form of acoustic or optical waves, such as those generated during radio wave and infrared data communications.

[0095] In one embodiment, various forms of media are involved in carrying one or more sequences of instructions to processor 304 for execution. For example, the instructions are initially executed on a disk or solid-state drive of a remote computer. The remote computer loads the instructions into its dynamic memory and sends the instructions over a telephone line using a modem. The local modem of computer system 300 receives the data on the telephone line and uses an infrared transmitter to convert the data to an infrared signal. An infrared detector receives the data carried in the infrared signal, and appropriate circuitry places the data on bus 302. Bus 302 carries the data to main memory 306, from which processor 304 retrieves and executes the instructions. The instructions received by main memory 306 may optionally be stored on storage device 310 before or after execution by processor 304.

[0096] Computer system 300 also includes a communication interface 318 connected to bus 302. Communication interface 318 provides two-way data communications coupled to a network link 320 connected to a local network 322. For example, communication interface 318 is an integrated services digital network (ISDN) card, a cable modem, a satellite modem, or a modem for providing a data communication connection with a corresponding type of telephone line. As another example, communication interface 318 is a local area network (LAN) card for providing a data communication connection with a compatible LAN. In some implementations, a wireless link is also implemented. In any such implementation, communication interface 318 sends and receives electrical, electromagnetic, or optical signals carrying digital data streams representing various information.

[0097] Network link 320 typically provides data communication to other data devices through one or more networks. For example, network link 320 provides a connection to a host computer 324 or to a cloud data center or device operated by an Internet Service Provider (ISP) 326 via a local network 322. ISP 326, in turn, provides data communication services via the worldwide packet data communication network now commonly referred to as the "Internet." Both local network 322 and Internet A 328 use electrical, electromagnetic, or optical signals that carry digital data streams. The signals passing through the various networks and the signals on network link 320 and through communication interface 318 are example forms of transmission media, where communication interface 318 carries digital data to and from computer system 300. In an embodiment, network link 320 comprises the aforementioned cloud 202 or a portion of cloud 202.

[0098] Computer system 300 sends messages and receives data, including program code, through network(s), network link 320, and communication interface 318. In an embodiment, computer system 300 receives code for processing. The received code is executed by processor 304 upon receipt and / or stored in storage device 310 or other non-volatile storage for later execution.

[0099] Autonomous Vehicle Architecture

[0100] Figure 4 Shown for autonomous vehicles (e.g., Figure 1 100). The architecture 400 includes a sensing module 402 (sometimes referred to as sensing circuitry), a planning module 404 (sometimes referred to as planning circuitry), a control module 406 (sometimes referred to as control circuitry), a positioning module 408 (sometimes referred to as positioning circuitry), and a database module 410 (sometimes referred to as database circuitry). Each module plays a role in the operation of the AV 100. Collectively, the modules 402, 404, 406, 408, and 410 may be Figure 1 406, 408, and 410 are each sometimes referred to as a processing circuit (e.g., computer hardware, computer software, or a combination thereof). A combination of any or all of modules 402, 404, 406, 408, and 410 is also an example of a processing circuit.

[0101] In use, the planning module 404 receives data representing a destination 412 and determines data representing a trajectory 414 (sometimes referred to as a route) that the AV 100 may travel in order to reach (e.g., arrive at) the destination 412. In order for the planning module 404 to determine the data representing the trajectory 414, the planning module 404 receives data from the perception module 402, the positioning module 408, and the database module 410.

[0102] The perception module 402 uses, for example, Figure 1 One or more sensors 121 are shown to identify nearby physical objects, classify the objects (e.g., into types such as pedestrians, bicycles, cars, traffic signs, etc.), and provide a scene description including the classified objects 416 to the planning module 404.

[0103] The planning module 404 also receives data representing the AV position 418 from the positioning module 408. The positioning module 408 determines the AV position by using data from the sensor 121 and data from the database module 410 (e.g., geographic data) to calculate the position. For example, the positioning module 408 uses data from a GNSS (Global Navigation Satellite System) sensor and geographic data to calculate the longitude and latitude of the AV. In an embodiment, the data used by the positioning module 408 includes a high-precision map with lane geometry, a map describing the road network connection properties, a map describing the physical properties of the lanes (such as traffic speed, traffic volume, the number of vehicle and bicycle lanes, lane width, lane traffic direction, or lane marking type and location, or a combination thereof), and a map describing the spatial location of road features (such as intersections, traffic signs, or various types of other driving signals, etc.). In an embodiment, the high-precision map is constructed by adding data to the low-precision map via automatic or manual annotation.

[0104] The control module 406 receives data representing the trajectory 414 and data representing the AV's position 418 and operates the AV's control functions 420 a - 420 c (e.g., steering, throttle, brakes, ignition) in a manner that will cause the AV 100 to travel the trajectory 414 to reach the destination 412. For example, if the trajectory 414 includes a left turn, the control module 406 will operate the control functions 420 a - 420 c in such a manner that the steering angle of the steering function will cause the AV 100 to turn left, and the throttle and brakes will cause the AV 100 to pause and wait for a passing pedestrian or vehicle before executing the turn.

[0105] Autonomous Vehicle Input

[0106] Figure 5 The perception module 402 ( Figure 4) used by the inputs 502a-502d (e.g., Figure 1 1) and outputs 504a-504d (e.g., sensor data). One input 502a is a LiDAR (Light Detection and Ranging) system (e.g., Figure 1 LiDAR 123 is shown. LiDAR is a technology that uses light (e.g., a beam of light such as infrared light) to obtain data about physical objects in its line of sight. The LiDAR system generates LiDAR data as output 504a. For example, LiDAR data is a collection of 3D or 2D points (also called a point cloud) used to construct a representation of the environment 190.

[0107] Another input 502b is a RADAR (radar) system. RADAR is a technology that uses radio waves to obtain data about nearby physical objects. RADAR can obtain data about objects that are not within the line of sight of the LiDAR system. RADAR system 502b generates RADAR data as output 504b. For example, RADAR data is one or more radio frequency electromagnetic signals used to construct a representation of environment 190.

[0108] Another input 502c is a camera system. The camera system uses one or more cameras (e.g., a digital camera using a light sensor such as a charge coupled device [CCD]) to acquire information about nearby physical objects. The camera system generates camera data as output 504c. The camera data is typically in the form of image data (e.g., data in an image data format such as RAW, JPEG, PNG, etc.). In some examples, the camera system has multiple independent cameras, such as for the purpose of stereo imaging (stereo vision), which enables the camera system to perceive depth. Although the objects perceived by the camera system are described here as "nearby", this is relative to the AV. In use, the camera system can be configured to "see" objects that are far away (e.g., up to 1 kilometer or more in front of the AV). Therefore, the camera system can have features such as sensors and lenses that are optimized for perceiving distant objects.

[0109] Another input 502d is a traffic light detection (TLD) system. The TLD system uses one or more cameras to obtain information about traffic lights, street signs, and other physical objects that provide visual navigation information. The TLD system generates TLD data as an output 504d. The TLD data often takes the form of image data (e.g., data in an image data format such as RAW, JPEG, PNG, etc.). A TLD system differs from a system that includes a camera in that the TLD system uses a camera with a wide field of view (e.g., using a wide-angle lens or a fisheye lens) to obtain information about as many physical objects that provide visual navigation information as possible, allowing the AV 100 to have access to all relevant navigation information provided by these objects. For example, the viewing angle of the TLD system can be approximately 120 degrees or more.

[0110] In some embodiments, the outputs 504a-504d are combined using sensor fusion techniques. Thus, the individual outputs 504a-504d are provided to other systems of the AV 100 (e.g., to a Figure 4 The combined output can be provided to other systems in the form of a planning module 404 shown in FIG. 4 , or in the form of a single combined output or multiple combined outputs of the same type (e.g., using the same combining technique or combining the same outputs, or both) or different types (e.g., using different individual combining techniques or combining different individual outputs, or both). In some embodiments, an early fusion technique is used. An early fusion technique is characterized in that the outputs are combined before one or more data processing steps are applied to the combined output. In some embodiments, a late fusion technique is used. A late fusion technique is characterized in that the outputs are combined after one or more data processing steps have been applied to the individual outputs.

[0111] Figure 6 An example of a LiDAR system 602 is shown (e.g., Figure 5 Input 502a is shown. The LiDAR system 602 emits light 604a-604c from a light emitter 606 (e.g., a laser emitter). The light emitted by the LiDAR system is typically not in the visible spectrum; for example, infrared light is often used. Some of the emitted light 604b encounters a physical object 608 (e.g., a vehicle) and reflects back to the LiDAR system 602. (The light emitted from the LiDAR system typically does not penetrate a physical object, such as a solid form of physical object.) The LiDAR system 602 also has one or more light detectors 610 for detecting the reflected light. In an embodiment, one or more data processing systems associated with the LiDAR system generate an image 612 representing a field of view 614 of the LiDAR system. The image 612 includes information representing the boundaries 616 of the physical object 608. In this way, the image 612 is used to determine the boundaries 616 of one or more physical objects in the vicinity of the AV.

[0112] Figure 7 LiDAR system 602 is shown in operation. In the scenario shown in this figure, AV 100 receives camera system output 504c in the form of image 702 and LiDAR system output 504a in the form of LiDAR data points 704. In use, the data processing system of AV 100 compares image 702 with data points 704. In particular, physical objects 706 identified in image 702 are also identified in data points 704. In this way, AV 100 perceives the boundaries of physical objects based on the contours and density of data points 704.

[0113] Figure 8 6 shows additional details of the operation of the LiDAR system 602. As described above, the AV 100 detects the boundaries of physical objects based on the characteristics of the data points detected by the LiDAR system 602. Figure 8 As shown, a flat object such as the ground 802 will reflect light 804a-804d emitted from the LiDAR system 602 in a consistent manner. In other words, because the LiDAR system 602 emits light at consistent intervals, the ground 802 will reflect light back to the LiDAR system 602 at the same consistent intervals. As the AV 100 drives over the ground 802, and nothing is blocking the road, the LiDAR system 602 will continue to detect light reflected from the next valid ground point 806. However, if an object 808 blocks the road, the light 804e-804f emitted by the LiDAR system 602 will reflect from points 810a-810b in a manner that does not conform to the expected consistent pattern. Based on this information, the AV 100 can determine that the object 808 is present.

[0114] Path Planning

[0115] Figure 9 Shown (for example, Figure 4 900 shows a block diagram of the relationship between the inputs and outputs of the planning module 404 (shown in FIG. 1 ). Typically, the output of the planning module 404 is a route 902 from a starting point 904 (e.g., a source location or initial location) to an end point 906 (e.g., a destination or final location). The route 902 is typically defined by one or more segments. For example, a segment is a distance to be traveled through at least a portion of a street, road, highway, lane, or other physical area suitable for automobile travel. In some examples, for example, if the AV 100 is an off-road vehicle such as a four-wheel drive (4WD) or all-wheel drive (AWD) car, SUV, or pickup truck, the route 902 includes "off-road" segments such as unpaved roads or open fields.

[0116] In addition to route 902, the planning module also outputs lane-level routing data 908. Lane-level routing data 908 is used to navigate the segments of route 902 at a specific time based on the conditions of those segments. For example, if route 902 comprises a multi-lane highway, lane-level routing data 908 includes trajectory planning data 910, which the AV 100 can use to select a lane from the multiple lanes based on, for example, whether an exit is imminent, whether other vehicles are present in one or more of the multiple lanes, or other factors that change over the course of a few minutes or less. Similarly, in some implementations, lane-level routing data 908 includes rate constraints 912 specific to a segment of route 902. For example, if the segment includes pedestrians or unexpected traffic, rate constraints 912 can limit the AV 100 to a slower travel speed than expected, such as a speed based on speed limit data for the segment.

[0117] In an embodiment, inputs to the planning module 404 include (e.g., Figure 4 ) database data 914, current location data 916 (e.g., Figure 4 AV position 418 shown), (e.g., for Figure 4 Destination data 918 and object data 920 (e.g., Figure 4 (See the example of a classified object 416 perceived by perception module 402, shown in Figure 4A). In some embodiments, database data 914 includes rules used during planning. Rules are specified using a formal language (e.g., using Boolean logic). In any given situation encountered by AV 100, at least some of these rules will apply to that situation. A rule applies to a given situation if it has conditions that are satisfied based on information available to AV 100 (e.g., information about the surrounding environment). Rules can have priorities. For example, a rule that says, "If the road is a freeway, move to the leftmost lane" may have a lower priority than a rule that says, "If an exit is within one mile, move to the rightmost lane."

[0118] Figure 10 As shown in the path planning (eg, by the planning module 404 ( Figure 4 )) uses a directed graph 1000. Typically, as Figure 10 A directed graph 1000, such as the one shown, is used to determine a path between any origin 1002 and destination 1004. In the real world, the distance separating the origin 1002 and destination 1004 may be relatively large (e.g., in two different metropolitan areas) or may be relatively small (e.g., two intersections adjacent to a city block or two lanes of a multi-lane road).

[0119] In one embodiment, directed graph 1000 includes nodes 1006a-1006d representing different locations that AV 100 may occupy between starting point 1002 and end point 1004. In some examples, for example, when starting point 1002 and end point 1004 represent different metropolitan areas, nodes 1006a-1006d represent sections of a road. In some examples, for example, when starting point 1002 and end point 1004 represent different locations on the same road, nodes 1006a-1006d represent different locations on that road. Thus, directed graph 1000 includes information at different levels of granularity. In one embodiment, a directed graph with high granularity is also a subgraph of another directed graph with a larger scale. For example, a directed graph where the start point 1002 and the end point 1004 are far apart (e.g., many miles apart) has most of its information at a low granularity and is based on stored data, but also includes some high granularity information for a portion of the directed graph that represents a physical location in the field of view of the AV 100.

[0120] Nodes 1006a-1006d are distinct from objects 1008a-1008b that cannot overlap with nodes. In an embodiment, at low granularity, objects 1008a-1008b represent areas where cars cannot drive through, such as areas without streets or roads. At high granularity, objects 1008a-1008b represent physical objects in the field of view of the AV 100, such as other cars, pedestrians, or other entities with which the AV 100 cannot share physical space. In an embodiment, some or all of objects 1008a-1008b are static objects (e.g., objects that do not change position, such as streetlights or utility poles) or dynamic objects (e.g., objects that can change position, such as pedestrians or other cars).

[0121] Nodes 1006a-1006d are connected by edges 1010a-1010c. If two nodes 1006a-1006b are connected by edge 1010a, the AV 100 can travel between one node 1006a and another node 1006b, for example, without having to travel to an intermediate node before reaching the other node 1006b. (When referring to the AV 100 traveling between nodes, it means that the AV 100 travels between two physical locations represented by the corresponding nodes.) Edges 1010a-1010c are typically bidirectional, in the sense that the AV 100 travels from a first node to a second node, or from a second node to a first node. In an embodiment, edges 1010a-1010c are unidirectional, in the sense that the AV 100 can travel from a first node to a second node, but the AV 100 cannot travel from a second node to the first node. Edges 1010a-1010c are unidirectional where they represent, for example, a one-way street, a single lane of a street, road, or highway, or other features that can only be traveled in one direction due to legal or physical constraints.

[0122] In an embodiment, the planning module 404 uses the directed graph 1000 to identify a path 1012 consisting of nodes and edges between the start point 1002 and the end point 1004 .

[0123] Edges 1010a-1010c have associated costs 1014a-1014b. Costs 1014a-1014b are values ​​representing the resources that would be expended if the AV 100 selected that edge. A typical resource is time. For example, if the physical distance represented by one edge 1010a is twice the physical distance represented by another edge 1010b, the cost 1014a associated with the first edge 1010a may be twice the cost 1014b associated with the second edge 1010b. Other factors that affect time include expected traffic, the number of intersections, speed limits, etc. Another typical resource is fuel economy. Two edges 1010a-1010b may represent the same physical distance, but one edge 1010a may require more fuel than the other edge 1010b, for example, due to road conditions, expected weather, etc.

[0124] When the planning module 404 identifies a path 1012 between the start point 1002 and the end point 1004 , the planning module 404 typically selects a path that is optimized for cost, eg, a path that has a minimum total cost when the individual costs of the edges are added together.

[0125] Autonomous vehicle control

[0126] Figure 11 Shown (for example, Figure 41 and 1 . A block diagram 1100 of the inputs and outputs of the control module 406 (shown in FIG. 1 ) is provided. The control module operates according to a controller 1102 that includes, for example: one or more processors similar to the processor 304 (e.g., one or more computing processors such as a microprocessor or microcontroller or both); short-term and / or long-term data storage devices similar to the main memory 306, ROM 308, and storage devices 310 (e.g., memory random access memory or flash memory or both); and instructions stored in the memory that, when executed (e.g., by the one or more processors), perform the operations of the controller 1102.

[0127] In an embodiment, the controller 1102 receives data representing a desired output 1104. The desired output 1104 typically includes speed, such as velocity and heading. The desired output 1104 may be based on, for example, Figure 4 104 ). Based on the desired output 1104, the controller 1102 generates data that can be used as a throttle input 1106 and a steering input 1108. The throttle input 1106 indicates the amount by which the throttle of the AV 100 (e.g., an acceleration control) should be engaged, for example, by engaging a steering pedal or engaging another throttle control, to achieve the desired output 1104. In some examples, the throttle input 1106 also includes data that can be used to engage the brakes of the AV 100 (e.g., a deceleration control). The steering input 1108 indicates the steering angle, for example, the angle at which the steering control of the AV (e.g., a steering wheel, a steering angle actuator, or other function for controlling the steering angle) should be positioned to achieve the desired output 1104.

[0128] In an embodiment, the controller 1102 receives feedback that is used when adjusting inputs to the throttle and steering. For example, if the AV 100 encounters a disturbance 1110, such as a hill, the AV 100's measured velocity 1112 drops below a desired output rate. In an embodiment, any measured output 1114 is provided to the controller 1102 so that any necessary adjustments can be made, for example, based on the difference 1113 between the measured velocity and the desired output. The measured outputs 1114 include measured position 1116, measured velocity 1118 (including velocity and heading), measured acceleration 1120, and other outputs measurable by the AV 100's sensors.

[0129] In an embodiment, information about the disturbance 1110 is pre-detected, for example, by a sensor such as a camera or a LiDAR sensor, and provided to the predictive feedback module 1122. The predictive feedback module 1122 then provides information to the controller 1102 that the controller 1102 can use to adjust accordingly. For example, if the sensors of the AV 100 detect ("see") a hill, the controller 1102 can use this information to prepare to engage the throttle at an appropriate time to avoid significant deceleration.

[0130] Figure 12 A block diagram 1200 illustrates the inputs, outputs, and components of the controller 1102. The controller 1102 includes a rate analyzer 1206 that influences the operation of a throttle / brake controller 1204. For example, the rate analyzer 1206 instructs the throttle / brake controller 1204 to accelerate or decelerate using the throttle / brake 1206 based on, for example, feedback received by the controller 1102 and processed by the rate analyzer 1202.

[0131] Controller 1102 also has a lateral tracking controller 1208 that affects the operation of a steering wheel controller 1210. For example, lateral tracking controller 1208 instructs steering wheel controller 1210 to adjust the position of a steering angle actuator 1212 based on, for example, feedback received by controller 1102 and processed by lateral tracking controller 1208.

[0132] The controller 1102 receives a number of inputs used to determine how to control the throttle / brakes 1206 and the steering angle actuator 1212. The planning module 404 provides information used by the controller 1102, for example, to select a heading for the AV 100 to begin operation and to determine which road segment to navigate when the AV 100 reaches an intersection. The positioning module 408 provides information describing the current location of the AV 100 to the controller 1102, for example, so that the controller 1102 can determine whether the AV 100 is at the location it would be expected to be based on how the throttle / brakes 1206 and the steering angle actuator 1212 are being controlled. In embodiments, the controller 1102 receives information from other inputs 1214, such as information received from a database, a computer network, or the like.

[0133] A simulation environment that improves the safety and reliability of AVs by reducing the generation of false positives and false negatives

[0134] Figure 13A computing environment 1302 is shown for generating simulation data used to generate metrics used by a computing device 1304 of the AV 100 to ensure the safety and reliability of the AV 100. The computing environment 1302 may include one or more processors 1306 that provide a simulation environment for the AV 100. In some implementations, the computing environment 1302 may be part of the computing environment 200, and the one or more processors 1306 may be part of the cloud data centers 204a, 204b, and 204c. The one or more processors 1306 may run a parameterizable instruction-level parallel (ILP) processor simulation acceleration engine (also referred to as a simulator), which may simulate the operation of the AV 100 based on data received from the AV 100 to generate simulation data. The one or more processors 1306 may use this simulation data to generate output metrics (also referred to as metrics), which may be used to update the programming of the computing device 1304 to ensure the safety and reliability of the AV 100.

[0135] Computing environment 1302 may include a database 1308 to store data indicating the operation of at least one module within computing device 1304 (e.g., perception module 402, planning module 404, control module 406, positioning module 408, database module 410, and / or any other module). One or more processors 1306 may receive the stored data from database 1308. One or more processors 1306 may generate a simulation of the operation of at least one module based on the received data to generate simulation data. Generating the simulation may involve representing the dynamic response of one or more modules of AV 100 by modeling the behavior of another system after the one or more modules of AV 100. In some implementations, the simulation may be a program that, when executed, replicates the functional relationships within and between different modules of AV 100, thereby performing a simulation of the behavior of AV 100. The results of the simulation are presented in the form of data (also referred to herein as simulation data). In some implementations, the simulation data can be updated to obtain simulation data that would have been generated by operation of the AV under other conditions (i.e., conditions different from the conditions under which the AV is currently operating), such as different traffic at various points in time, different weather conditions, and / or any other different conditions with different driving performance. One or more of the processors 1306 can store the simulation data in the database 1310.

[0136] One or more of the processors 1306 can identify at least a portion of the simulation that indicates a deviation between the collected data and the simulated operation of the AV 100. The deviation can represent a false positive or false negative generated by the computing device 1304 during the operation of the AV 100. In one implementation, the one or more processors 1306 can identify the portion with the deviation as follows. The one or more processors 1306 can feed a first signal comprising time-stamped collected data and a second signal comprising time-stamped simulated data (with the same time stamp as the collected data to generate an accurate comparison) to a comparator, which can generate a comparison indicating the specific portion with the deviation. In some implementations, the deviation can be detected when the deviation from such a comparison exceeds a threshold. In some implementations, the comparator can be a hardware electronic device coupled to the one or more processors 1306. In some implementations, the comparator can be software (e.g., a software application) communicatively coupled to the one or more processors. The identified portion of the simulation can be simulation data over a period of time, which can be a preset time (e.g., 5 seconds, 15 seconds, 30 seconds, 1 minute or 2 minutes, etc.) before a specific event (e.g., an alarm is generated or a collision occurs) and another preset time (e.g., 5 seconds, 15 seconds, 30 seconds, 1 minute or 2 minutes, etc.) after the specific event.

[0137] One or more of the processors 1306 may analyze at least one portion of the simulation to generate metrics for the at least one portion of the simulation. In some examples, the metrics may include specific brake control commands and debugging data. The brake control commands may include details on when the brakes of the AV 100 should be applied. For example, the brake control commands may specify that the brakes should be applied if the AV 100 is operating and the AV 100 behaves or acts in a specific manner (e.g., if the sensor data, trajectory data, and vehicle status of the AV 100 have a preset value or a value within a preset range). The debugging data may include programming instructions or details that the computing device may execute to identify instances where discrepancies between the collected data and the simulated data are likely to occur and to prevent false positives from being generated at these instances.

[0138] In some implementations, the one or more processors 1306 may send the metrics to the computing device 1304, which may be programmed to use the metrics to generate future instructions for at least one module to avoid further deviations between the collected data and the simulated operation of the AV 100. In other implementations, instead of sending the metrics to the computing device 1304, the one or more processors 1306 may use the metrics to generate future instructions for at least one module and send data representing only those instructions to the computing device 1304, which may execute the instructions to avoid further deviations between the collected data and the simulated operation of the AV 100. The one or more processors 1306 may store the metrics in a database 1312.

[0139] As described above, a deviation may represent a false positive generated by computing device 1304, where the collected data (stored in database 1308) indicates that computing device 1304 generated an alert of a potential collision, while simulated data (stored in database 1310) indicates that the alert was false. A deviation may represent a false negative generated by computing device 1304, where the collected data (stored in database 1308) indicates that computing device 1304 failed to indicate an alert of a potential collision, while simulated data (stored in database 1310) indicates a potential collision. In some implementations, the alert may be the activation of one or more lights or audio signals connected to computing device 1304 of AV 100. The alert may be designed to indicate an unexpected occurrence in the vicinity of AV 100. In some implementations, the alert may additionally or alternatively be any type of notification to the driver of AV 100 indicating the occurrence of an unexpected occurrence (such as an email, text message, or social media notification).

[0140] An unexpected occurrence can be the unexpected presence of various factors that were not or cannot be identified before planning the trajectory of the AV 100. These factors can include one or more objects or one or more living beings in the path of the AV 100 (e.g., the planned trajectory), or unexpected weather conditions encountered during the operation of the AV 100. These objects in the path of the AV 100 can be other vehicles, buildings, or construction signs, etc. Living beings in the path of the AV 100 can be humans (e.g., pedestrians, drivers of other vehicles, or any other humans) or animals (e.g., galloping animals or wild animals). An unexpected occurrence can also include unexpected congestion caused by various unexpected factors such as: traffic jams, sudden (e.g., within a threshold time) surges in traffic, improper parking of one or more other vehicles, lane closures, road narrowings, accidents, lane changes, traffic signal malfunctions, and / or any other one or more factors in the planned trajectory of the AV 100.

[0141] Although the alert is described above as being designed to indicate an unexpected occurrence in the vicinity of the AV 100, in some implementations, the alert may include a notification indicating a malfunction of the computing device 1304. The malfunction of the computing device 1304 may include one or more of the following: at least one computer or processor within the computing device 1304 is not powered on; at least one computer or processor within the computing device 1304 stops functioning; the computing device 1304 fails to receive power or stops charging; at least one computer or processor within the computing device 1304 stops communicating with a screen (e.g., a graphical user interface) within the AV 100, causing the screen to become malfunctioning or blank; an operating system of the computing device 1304 fails; at least one module on the computing device 1304 (e.g., the perception module 402, the planning module 404, the control module 406, the positioning module 408, the database module 410, and / or any other module) fails; and / or the computing device 1304 fails to connect to a communication network connecting the computing device 1304 to the one or more processors 1306; and / or the computing device 1304 fails to connect to a communication network connecting the computing device 1304 to the one or more processors 1306; and / or the like.

[0142] In some implementations, one or more processors 1306 may be coupled to a computing system 1314, which may be a computer accessible to a user (e.g., a test administrator). Computing system 1314 may present (e.g., display on a graphical user interface 1316) a plurality of AVs from which a user (e.g., a system administrator or test engineer, etc.) may select one or more AVs (e.g., AV 100) to be simulated to generate metrics to improve the safety and reliability of the selected one or more AVs (e.g., AV 100). In some implementations, computing system 1314 runs a software application that allows the user to interactively make such selections. Such a software application may be served on the backend by one or more processors 1306 (e.g., such a software application may be served on the backend by a simulator, such that the software application is a simulation software application that allows the user to select the AV to be simulated). Once AV 100 is selected on such a software application, one or more processors 1306 may collect data from computing device 1304 to initiate the simulation. In some implementations, the computing system 1314 can present (e.g., display on a graphical user interface 1316) one or more options to change the conditions of the simulation data (i.e., conditions that are different from the conditions under which the AV is operating). The different conditions can be different weather conditions, different traffic at a different time, and / or any other different conditions.

[0143] Although manual selection of the AV 100 from among several AVs is described, in some implementations, the selection of the AV 100 can be automatic. For example, the one or more processors 1306 can select the AVs in a fixed order (e.g., the first AV is first, the second AV is second, and the third AV is third, etc.). In some implementations, the one or more processors 1306 can select the AVs based on their safety. For example, the AVs can be selected in order of increasing safety, i.e., the least safe AV can be selected first (because it may be more important to ensure that the AV is safe and reliable first), and the safest AV can be selected last (because it may be less important to ensure that the AV is safe). The safety of any AV can be measured by the number of events (e.g., accidents) within a threshold amount of time (e.g., the past week, month, year, or any other time period, as measured from the current time or another preset time).

[0144] Additionally or alternatively, the one or more processors 1306 may automatically select an AV based on the reliability of the AV. For example, the AVs may be selected in order of increasing reliability, i.e., the least reliable AV may be selected first (because it may be more important to ensure that the AV is safe and reliable first), and the most reliable AV may be selected last (because it may be less important to ensure that the AV is reliable). The reliability of the AV may be measured by the number of false negatives (as described above) and / or the number of false positives (also described above) generated by the computing device of the AV (e.g., the computing device 1304 of the AV 100).

[0145] As described above, the identification of false positives and false negatives can be used to generate metrics to prevent the computing device 1304 from generating such false positives or false negatives. Preventing false positives can reduce (e.g., minimize) the number of false alerts of impending collisions generated by the computing device 1304. This reduction in false alerts can reduce the number of unnecessary interventions required (e.g., by the driver or remote controller) during operation of the AV 100. This enhances the reliability of the AV 100 and instills confidence in the driver's operation of the AV 100. Furthermore, preventing false negatives can ensure that the computing device 1304 generates appropriate alerts indicating potential collisions. This can advantageously prevent collisions. This collision prevention can enhance the safety and reliability of the AV 100.

[0146] Figure 14A process for making the AV 100 safe and reliable by reducing false positives and false negatives generated by the computing device 1304 of the AV 100 is shown. One or more of the processors 1306 can receive data collected by the computing device 1304 during operation of the AV 100 at 1402. In one implementation, one or more of the processors 1306 can receive the data directly from the computing device 1304 (which can be in real time). In another implementation, one or more of the processors 1306 can receive the data from a database 1308 that has received such data and store the data.

[0147] At 1404, one or more of the processors 1306 may generate a simulation of the operation of at least one module based on the received data to generate simulation data. Generating the simulation may involve representing the dynamic response of one or more modules of the AV 100 by modeling the behavior of another system after the one or more modules of the AV 100. The simulation may be a program that, when executed, replicates the functional relationships within and between the different modules of the AV 100, thereby performing a simulation of the behavior of the AV 100. The results of the simulation are presented in the form of data (also referred to herein as simulation data). One or more of the processors 1306 may store the simulation data in a database 1310.

[0148] One or more of the processors 1306 may identify, at 1406, at least a portion of the simulation that indicates a deviation between the collected data and the simulated data of the AV 100. Such a deviation may represent a false positive or false negative generated by the computing device 1304 during operation of the AV 100. In some examples, the deviation may represent a false positive generated by the computing device 1304, where the collected data (stored in the database 1308) indicates that the computing device 1304 generated an alert of a potential collision, while the simulated data (stored in the database 1310) indicates that the alert was false. In some implementations, the deviation may represent a false negative generated by the computing device 1304, where the collected data (stored in the database 1308) indicates that the computing device 1304 failed to indicate an alert of a potential collision, while the simulated data (stored in the database 1310) indicates a potential collision.

[0149] At 1408, one or more of the processors 1306 may analyze at least one portion of the simulation to generate metrics for the at least one portion of the simulation. In some examples, the metrics may include specific brake control commands and debugging data. The brake control commands may include details on when the brakes of the AV 100 should be applied. For example, the brake control commands may specify that the brakes should be applied if the AV 100 is operating and the AV 100 behaves or acts in a specific manner (e.g., if the sensor data, trajectory data, and vehicle status of the AV 100 have a preset value or a value within a preset range). The debugging data may include programming instructions or details that the computing device may execute to identify instances where discrepancies between the collected data and the simulated data are likely to occur and to prevent false positives from being generated at these instances.

[0150] The one or more processors 1306 may send the metrics at 1410 to the computing device 1304, which may be programmed to use the metrics to generate future instructions for at least one module to avoid further deviations between the collected data and the simulated operation of the AV 100. In some implementations, instead of sending the metrics to the computing device 1304, the one or more processors 1306 may use the metrics to generate future instructions for at least one module and send data representing only those instructions to the computing device 1304, which may execute those instructions to avoid further deviations between the collected data and the simulated operation of the AV 100. The one or more processors 1306 may store the metrics in a database 1312.

[0151] Figure 15 13. A specific example of avoiding false alarms generated by the computing device 1304 is shown. The data received from the computing device 1304 of the AV 100 includes sensor data 1502, trajectory data 1504, and vehicle state 1506. The sensor data 1502 may be data sensed by the sensor 121 (and may include any other data related to the sensor 121 as described above). The trajectory data 1504 may be data representing the trajectory 198 (and may include any other data related to the trajectory 198 as described above). The vehicle state 1506 may be a state or condition of the AV 100, such as the AV's position, linear and angular velocity and acceleration, and heading (e.g., the orientation of the front end of the AV 100).

[0152] One or more processors 1306 may generate a simulation 1508 of a module within the AV 100. In the example shown, the module is an emergency brake module, which may be part of the control module 406 that controls the braking of the AV 100. Generating the simulation may involve representing the dynamic response of the emergency brake module by modeling the behavior of another system after the emergency brake module of the AV 100. In some implementations, the simulation may be a program that, when executed, replicates the functional relationships within the emergency brake module, thereby executing a simulated version of the behavior of the emergency brake module. The results of the simulation of the emergency brake module are presented as simulation data. The simulation data may be generated so that it corresponds to the collected data (here, sensor data 1502, trajectory data 1504, and vehicle state 1506) (the simulation data is generated for the same time as the collected data), thereby enabling accurate comparison of the collected data with the simulation data. The one or more processors 1306 may store the simulation data in a database 1310.

[0153] The one or more processors 1306 can compare the collected data (stored in the database 1308) with the simulated data (stored in the database 1310) to identify instances (e.g., time periods) where there is a discrepancy between the collected data and the simulated data. Here, a discrepancy can be a situation where the collected data (i.e., the sensor data 1502, the trajectory data 1504, and the vehicle state 1506) indicates that the computing device 1304 has generated an alert of a potential collision, while the corresponding simulated data (i.e., the simulated data analyzed at a time point corresponding to the time point of the collected data) indicates that the alert is false (i.e., there is no possibility of a collision, or such a possibility is below a threshold).

[0154] The one or more processors 1306 can evaluate instances where deviations exist to generate metrics to prevent future false alarms. Such metrics can include specific brake control commands 1510 and debugging data 1512. The brake control commands 1510 can include details on when the brakes of the AV 100 should be applied. For example, the brake control commands 1510 can specify that the brakes should be applied when the sensor data, trajectory data, and vehicle status have a preset value (or a value within a preset range). The debugging data 1512 can include programming instructions or details that the computing device 1304 can execute to identify instances where deviations between the collected data and the simulated data are likely to occur and to prevent false alarms from being generated at these instances.

[0155] The one or more processors 1306 may store the metrics in a database 1312. The one or more processors 1306 may then send the stored metrics to the computing device 1304. In other implementations, the one or more processors 1306 may send the metrics directly to the computing device 1304 (i.e., before storing them). These metrics include specific instructions (e.g., brake control commands and debugging data) to be executed by the emergency brake module, which is part of the control module 406 that controls the braking of the AV 100, to avoid false alarms.

[0156] Figure 16 A specific example of avoiding false negatives from the computing device 1304 of the AV 100 is shown. One or more processors 1306 may receive data collected by the computing device 1304 during operation of the AV 100. Here, the data collected by the computing device 1304 is an initial vehicle state 1604. The initial vehicle state 1604 may be a state or condition of the AV 100, such as the position, linear and angular velocity and acceleration, and heading (e.g., the orientation of the front end of the AV 100) of the AV 100. In one implementation, the one or more processors 1306 may receive the initial vehicle state 1604 from the database 1308, which has received and stored such initial vehicle state 1604 as shown. In another implementation, the one or more processors 1306 may receive the initial vehicle state 1604 directly from the computing device 1304 of the AV 100 (which may be in real time).

[0157] One or more of the processors 1306 can generate simulations of the operation of various modules based on the received vehicle state, such as simulation 1606 of the planning module 404, simulation 1608 of the control module 406, and simulation 1508 of the emergency brake module 1508. Simulation 1606 can receive a track of an object 1610 from database 1308. The track of object 1610 is the trajectory of the object and can indicate the location of the object over time. Simulations 1606, 1608, and 1508 can generate simulation data (e.g., simulation 1606 of the planning module 404 can generate a simulated planned trajectory 1612, and simulation 1608 of the control module can generate control commands 1613).

[0158] The simulations 1606, 1608, and 1508 of the various modules may generate corresponding simulation data including a simulated vehicle state 1614. The initial vehicle state 1604 and the simulated vehicle state 1614 may be fed into a comparator 1616, which may perform a comparison to identify at least a portion of the time-stamped simulation data that indicates a deviation between the collected data and the simulated data of the AV 100. The deviation may represent a false negative generated by the computing device 1304, where the initial vehicle state 1604 indicates that the computing device 1304 failed to indicate an alert for a potential collision, while the simulated vehicle state 1614 indicates a potential collision.

[0159] One or more of the processors 1306 can analyze the portion of simulation data identified by the comparator 1616 to generate metrics for the at least one portion of the simulation. If the different data received by the transformation module 1622 originally had different reference coordinates, the transformation module 1622 can perform a coordinate transformation on the data. These metrics can include safety control commands 1618. Safety control commands 1618 can include, for example, brake control commands 1510. Brake control commands 1510 can include details on when the brakes of the AV 100 should be applied. For example, brake control commands 1510 can specify that the brakes should be applied when the sensor data, trajectory data, and vehicle status have a preset value (or a value within a preset range). Control commands 1613 and 1618 pass through a multiplexer 1620 for processing before being further used by the one or more processors 1306 to continue the simulation.

[0160] The one or more processors 1306 may send the generated metrics to the computing device 1304, which may be programmed to use the metrics to generate future instructions for the at least one module to avoid further deviations between the collected data and the simulated operation of the AV 100. In some implementations, instead of sending the metrics to the computing device 1304, the one or more processors 1306 may use the metrics to generate future instructions for the at least one module and send data representing only those instructions to the computing device 1304, which may execute the instructions to avoid further deviations between the collected data and the simulated operation of the AV 100.

[0161] To improve the safety and reliability of AVs by increasing the accuracy of future alerts indicating another potential collision Sexual simulation environment

[0162] On the other hand, Figure 13The architecture and components within the architecture may be used to generate a simulation of the AV 100, examine the simulation data to identify a portion of the simulation when a takeover from an automated driving mode occurs in response to a computing device generating an alert indicating a potential collision, analyze the portion of the simulation to determine a metric indicating whether a collision would have occurred if the takeover had not occurred (i.e., a probability of such a collision occurring is greater than a threshold), and use such metric to improve the accuracy of a future alert indicating another potential collision.

[0163] In this regard, one or more processors 1306 may receive data collected by computing device 1304 during operation of AV 100. One or more processors 1306 may store the collected data in database 1308. One or more processors 1306 may generate simulations of the operation of at least some modules of AV 100 (e.g., a planning module and a control module) based on the collected data. Generating the simulations may involve representing the dynamic responses of at least one module of AV 100 by modeling the behavior of another system after the behavior of at least one module of AV 100. In some implementations, the simulation may be a program that, when executed, replicates the functional relationships within and between different modules of AV 100, thereby performing a simulation of the behavior of AV 100. The results of the simulation are presented in the form of data (also referred to herein as simulation data). One or more processors 1306 may store the simulation data in database 1310.

[0164] The one or more processors 1306 can identify at least a portion of the simulation that indicates a takeover (e.g., by a driver or a remote entity) from the autonomous driving mode in response to the computing device 1304 generating an alert indicating a potential collision. To make such an identification, the one or more processors 1306 can determine a point in time for the takeover. In some implementations, the point in time for the takeover can be determined as the time when the computing device 1304 generates the associated alert. The identified portion of the simulation can be simulation data over a period of time, which can be a preset time (e.g., 5 seconds, 15 seconds, 30 seconds, 1 minute, or 2 minutes, etc.) before a particular event (e.g., when the takeover occurred or the associated alert was generated) and another preset time (e.g., 5 seconds, 15 seconds, 30 seconds, 1 minute, or 2 minutes, etc.) after the particular event.

[0165] In some implementations, the alert generated during the takeover can be the activation of one or more lights or audio on the computing device 1304 connected to the AV 100. The alert can be designed to indicate an unexpected occurrence in the vicinity of the AV 100. In some implementations, the alert can additionally or alternatively be any type of notification to the driver of the AV 100 indicating the occurrence of an unexpected event (such as an email, text message, or social media notification, etc.).

[0166] When an alarm is generated, a takeover can be performed by a driver (e.g., a safety driver) within the AV 100. The driver can take over by pressing the emergency brake. The takeover can include navigating the AV 100 to a safe location away from the vicinity of the AV 100. The unexpected occurrence can include one or more of the following: an unexpected blockage in the path the AV 100 is traversing, an object or one or more creatures unexpectedly being in the path of the AV 100, and unexpected weather conditions being experienced during operation of the AV 100.

[0167] Although the takeover is described as being performed by a driver (e.g., a safety driver), in some implementations, the takeover may be performed remotely by a human user or an automated backend computer system configured to control one or more modules (e.g., control module 406 controlling the braking of the AV 100). In these implementations, an alarm may also be triggered by the computing device 1304 determining (e.g., detecting) a change in a driver's vital sign exceeding a threshold. A vital sign may be pulse rate, respiratory rate, blood pressure, or any other vital sign that may be affected by an impending collision. Such an alarm may initiate a remote takeover to prevent a collision. For example, if the driver's pulse rate (as a measure of heart rate or beats per minute) is determined to exceed 120 beats per minute (which is significantly higher than the normal range of 60 to 100 beats per minute for an adult), a remote takeover may be initiated. In some examples, if the driver's respiratory rate (i.e., the number of breaths a person takes per minute) is determined to exceed 20 breaths per minute (which is significantly higher than the normal range of 12 to 16 breaths per minute for an adult), a remote takeover may be initiated. In some examples, remote takeover may be initiated if the blood pressure (i.e., the force of blood pushing against arterial walls during contraction and relaxation of the heart) of another normal driver (i.e., a driver with a systolic blood pressure of less than 120 and a diastolic blood pressure of less than 80) is determined to have a Grade 1 blood pressure (i.e., a systolic blood pressure of 130 to 139 or a diastolic blood pressure of 80 to 89) or a Grade 2 blood pressure (i.e., a systolic blood pressure of 140 or higher or a diastolic blood pressure of 90 or higher).

[0168] The one or more processors 1306 can analyze at least a portion of the simulation data to determine metrics indicating whether a collision would have occurred if the takeover had not occurred and control commands to avoid such a collision. For example, the metrics can include specific brake control commands that can include details on when the brakes of the AV 100 should be applied so that a collision can be avoided in the absence of the takeover. For example, the brake control commands can specify that the brakes should be applied if the AV 100 is operating and the AV 100 behaves or acts in a specific manner (e.g., if the sensor data, trajectory data, and / or vehicle state of the AV 100 have a preset value or a value within a preset range). In some implementations, the metrics can also include debugging data that can include programming instructions or details that the computing device can execute to identify instances in which a collision would have occurred if the takeover had not occurred and to prevent such a collision.

[0169] In some implementations, the one or more processors 1306 may send the metrics to the computing device 1304, which may be programmed to use the metrics to generate programmed instructions to improve the accuracy of future alerts indicating another potential collision. In other implementations, instead of sending the metrics to the computing device 1304, the one or more processors 1306 may use the metrics to generate instructions and send data representing the instructions to the computing device 1304, which may execute the instructions to improve the accuracy of future alerts indicating another potential collision. The one or more processors 1306 may store the metrics in a database 1312.

[0170] The above implementations are advantageous. For example, as described above, these metrics can be used to improve the accuracy of future alerts (indicating another potential collision). This improvement can, in turn, reduce the number of collisions experienced by the AV 100, thereby making the AV 100 safer and more reliable. Furthermore, in some implementations, the alert can initiate automatic takeover of the remote system, which can be advantageous in situations where there is no human driver, or where the human driver is negligent, careless, and / or suffering from a medical condition (e.g., significant deviations from normal values ​​for vital signs, such as when the driver is experiencing a heart attack). This further improves the reliability of the AV 100.

[0171] Figure 17A process for generating metrics to improve the accuracy of future alerts indicating another potential collision is shown. At 1702, one or more processors 1306 may receive data collected by computing device 1304 during operation of AV 100. One or more processors 1306 may store the received data in database 1308. At 1704, one or more processors 1306 may generate a simulation of the operation of AV 100 based on the received data to generate simulation data. One or more processors 1306 may store the simulation data in database 1310. At 1706, one or more processors 1306 may identify at least a portion of the simulation that indicates a takeover (e.g., by a driver or a remote entity) from an autonomous driving mode in response to computing device 1304 generating an alert indicating a potential collision. One or more processors 1306 may make such an identification using signal processing techniques, such as event-based analysis of video and / or images. In some implementations, the alert may be the activation of one or more lights or audio signals on computing device 1304 connected to AV 100. The alert may be designed to indicate an unexpected occurrence in the vicinity of the AV 100. In some implementations, the alert may additionally or alternatively be any type of notification to the driver of the AV 100 indicating the unexpected occurrence (such as an email, text message, or social media notification, etc.).

[0172] When an alarm is generated, a takeover can be performed by a driver (e.g., a safety driver) within the AV 100. The driver can take over by pressing the emergency brake. The takeover can include navigating the AV 100 to a safe location away from the vicinity of the AV 100. The unexpected occurrence can include one or more of the following: an unexpected blockage in the path the AV 100 is traversing, an object or one or more creatures unexpectedly being in the path of the AV 100, and unexpected weather conditions being experienced during operation of the AV 100.

[0173] Although the takeover is described as being performed by a driver (e.g., a safety driver), in some implementations, the takeover may be performed remotely by a human user or an automated backend computer system configured to control one or more modules (e.g., control module 406 controlling the braking of the AV 100). In these implementations, an alarm may also be triggered by the computing device 1304 determining (e.g., detecting) a change in a driver's vital sign exceeding a threshold. A vital sign may be pulse rate, respiratory rate, blood pressure, or any other vital sign that may be affected by an impending collision. Such an alarm may initiate a remote takeover to prevent a collision. For example, if the driver's pulse rate (as a measure of heart rate or beats per minute) is determined to exceed 120 beats per minute (which is significantly higher than the normal range of 60 to 100 beats per minute for an adult), a remote takeover may be initiated. In some examples, if the driver's respiratory rate (i.e., the number of breaths a person takes per minute) is determined to exceed 20 breaths per minute (which is significantly higher than the normal range of 12 to 16 breaths per minute for an adult), a remote takeover may be initiated. In some examples, remote takeover may be initiated if the blood pressure (i.e., the force of blood pushing against arterial walls during contraction and relaxation of the heart) of another normal driver (i.e., a driver with a systolic blood pressure of less than 120 and a diastolic blood pressure of less than 80) is determined to have a Grade 1 blood pressure (i.e., a systolic blood pressure of 130 to 139 or a diastolic blood pressure of 80 to 89) or a Grade 2 blood pressure (i.e., a systolic blood pressure of 140 or higher or a diastolic blood pressure of 90 or higher).

[0174] At 1708, the one or more processors 1306 may analyze at least a portion of the simulation data to determine metrics indicating whether a collision would have occurred if the takeover had not occurred and control commands to avoid such a collision. For example, the metrics may include specific braking control commands that may include details on when the brakes of the AV 100 should be applied so that a collision that would have occurred in the absence of the takeover can be avoided. For example, the braking control commands may specify that the brakes should be applied if the AV 100 is operating and the AV 100 behaves or acts in a specific manner (e.g., if the sensor data, trajectory data, and / or vehicle state of the AV 100 have a preset value or a value within a preset range). In some implementations, the metrics may also include debugging data that may include programming instructions or details that the computing device may execute to identify instances in which a collision would have occurred if the takeover had not occurred and to prevent such a collision.

[0175] In some implementations, the one or more processors 1306 may send the metrics at 1710 to the computing device 1304, which may be programmed to use the metrics to generate future instructions to improve the accuracy of future alerts indicating another potential collision. In other implementations, instead of sending the metrics to the computing device 1304, the one or more processors 1306 may use the metrics to generate instructions and send data representing the instructions to the computing device 1304, which may execute the instructions to improve the accuracy of future alerts indicating another potential collision. The one or more processors 1306 may store the metrics in a database 1312.

[0176] The above implementations are advantageous. For example, as described above, these metrics can be used to improve the accuracy of future alerts (indicating another potential collision). This improvement can, in turn, reduce the number of collisions experienced by the AV 100, thereby making the AV 100 safer and more reliable. Furthermore, in some implementations, the alert can initiate automatic takeover of the remote system, which can be advantageous in situations where there is no human driver, or where the human driver is negligent, careless, and / or suffering from a medical condition (e.g., significant deviations from normal values ​​for vital signs, such as when the driver is experiencing a heart attack). This further improves the reliability of the AV 100.

[0177] Figure 18 13. The process of generating metrics to improve the accuracy of future warnings indicating another potential collision is shown. The one or more processors 1306 can receive data collected by the computing device 1304 during operation of the AV 100 (e.g., the initial vehicle state 1604) from the database 1308. In an alternative implementation, the one or more processors 1306 can receive the initial vehicle state 1604 directly from the computing device 1304 of the AV 100. The initial vehicle state 1604 can be a state or condition of the AV 100, such as the position, linear and angular velocity and acceleration, and heading (e.g., the orientation of the front end of the AV 100) of the AV 100.

[0178] The one or more processors 1306 may generate simulations 1606 and 1608 of the operations of the planning module 404 and the control module 406 of the AV 100 based on the received data to generate simulation data. Simulation 1606 may receive a track of an object 1610 from database 1308. The track of object 1610 is the trajectory of the object and may indicate the location of the object over time. Simulations 1606 and 1608 may generate simulation data (e.g., simulation 1606 of planning module 404 may generate a simulated planned trajectory 1612, and simulation 1608 of control module 406 may generate control commands 1613). Simulations 1606 and 1608 of each module may generate corresponding simulation data, including simulated vehicle states 1614. The one or more processors 1306 may provide the simulation data to update simulation 1606 of planning module 404.

[0179] The one or more processors 1306 can identify at least a portion of the simulated data that indicates a takeover from the autonomous driving mode (e.g., by a driver or a remote entity) in response to the computing device 1304 generating an alert indicating a potential collision. In some implementations, the alert can be an activation of one or more lights or audio signals on the computing device 1304 connected to the AV 100. The alert can be designed to indicate an unexpected occurrence in the vicinity of the AV 100. In some implementations, the alert can additionally or alternatively be any type of notification to the driver of the AV 100 indicating an unexpected occurrence (such as an email, text message, or social media notification, etc.).

[0180] When an alarm is generated, a takeover can be performed by a driver (e.g., a safety driver) within the AV 100. The driver can take over by pressing the emergency brake. The takeover can include navigating the AV 100 to a safe location away from the vicinity of the AV 100. The unexpected occurrence can include one or more of the following: an unexpected blockage in the path the AV 100 is traversing, an object or one or more creatures unexpectedly being in the path of the AV 100, and unexpected weather conditions being experienced during operation of the AV 100.

[0181] Although the takeover is described as being performed by a driver (e.g., a safety driver), in some implementations, the takeover may be performed remotely by a human user or an automated backend computer system configured to control one or more modules (e.g., control module 406 controlling the braking of the AV 100). In these implementations, an alarm may also be triggered by the computing device 1304 determining (e.g., detecting) a change in a driver's vital sign exceeding a threshold. A vital sign may be pulse rate, respiratory rate, blood pressure, or any other vital sign that may be affected by an impending collision. Such an alarm may initiate a remote takeover to prevent a collision. For example, if the driver's pulse rate (as a measure of heart rate or beats per minute) is determined to exceed 120 beats per minute (which is significantly higher than the normal range of 60 to 100 beats per minute for an adult), a remote takeover may be initiated. In some examples, if the driver's respiratory rate (i.e., the number of breaths a person takes per minute) is determined to exceed 20 breaths per minute (which is significantly higher than the normal range of 12 to 16 breaths per minute for an adult), a remote takeover may be initiated. In some examples, remote takeover may be initiated if the blood pressure (i.e., the force of blood pushing against arterial walls during contraction and relaxation of the heart) of another normal driver (i.e., a driver with a systolic blood pressure of less than 120 and a diastolic blood pressure of less than 80) is determined to have a Grade 1 blood pressure (i.e., a systolic blood pressure of 130 to 139 or a diastolic blood pressure of 80 to 89) or a Grade 2 blood pressure (i.e., a systolic blood pressure of 140 or higher or a diastolic blood pressure of 90 or higher).

[0182] The one or more processors 1306 may analyze at least a portion of the simulation data at 1708 to determine metrics indicating whether a collision would have occurred if the takeover had not occurred and control commands to avoid such a collision. For example, the metrics may include specific braking control commands that may include details on when the brakes of the AV 100 should be applied so that a collision could have been avoided in the absence of the takeover. For example, the braking control commands may specify that the brakes should be applied if the AV 100 is operating and the AV 100 behaves or acts in a specific manner (e.g., if the sensor data, trajectory data, and / or vehicle state of the AV 100 has a preset value or a value within a preset range). In some implementations, the metrics may also include debugging data that may include programming instructions or details that the computing device may execute to identify instances in which a collision would have occurred if the takeover had not occurred and to prevent such a collision.

[0183] In some implementations, the one or more processors 1306 may send the metrics at 1710 to the computing device 1304, which may be programmed to use the metrics to generate future instructions to improve the accuracy of future alerts indicating another potential collision. In other implementations, instead of sending the metrics to the computing device 1304, the one or more processors 1306 may use the metrics to generate instructions and send data representing the instructions to the computing device 1304, which may execute the instructions to improve the accuracy of future alerts indicating another potential collision. The one or more processors 1306 may store the metrics in a database 1312.

[0184] The implementation and all functional operations described in this specification can be implemented in digital electronic circuitry, or in computer software, firmware, or hardware including the structures disclosed in this specification and their structural equivalents, or a combination of one or more of these. Implementations can be implemented as one or more computer program products, i.e., one or more modules of computer program instructions encoded on a computer-readable medium for execution by or to control the operation of a data processing device. The computer-readable medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a composition of matter that implements a machine-readable propagated signal, or a combination of one or more of these. The term "computing system" includes all devices, apparatuses, and machines for processing data, including, for example, a programmable processor, a computer, or multiple processors or computers. In addition to hardware, a device may also include code that creates an execution environment for the computer program in question, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of these. A propagated signal is an artificially generated signal, such as a machine-generated electrical, optical, or electromagnetic signal generated to encode information for transmission to a suitable receiver device.

[0185] A computer program (also known as a program, software, software application, script, or code) can be written in any suitable form of a programming language including a compiled language or an interpreted language, and can be deployed in any suitable form including components, subroutines, or other units suitable for use in a computing environment as a stand-alone program or as a module. A computer program does not necessarily correspond to a file in a file system. A program can be stored in: a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), a dedicated single file for the program in question, or multiple collaborative files (e.g., files that store one or more modules, subroutines, or partial codes). A computer program can be deployed to execute on a computer or multiple computers that are located at a site or distributed between multiple sites and interconnected via a communication network.

[0186] The processes and logic flows described in this specification can be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by a device, and the device can be implemented as a special-purpose logic circuit such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit).

[0187] Processors suitable for executing computer programs include, for example, both general-purpose and special-purpose microprocessors, as well as any one or more processors of any suitable type of digital computer. Typically, a processor will receive instructions and data from read-only memory or random access memory, or both. Components of a computer may include a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer may also include one or more mass storage devices, such as magnetic disks, magneto-optical disks, or optical disks, for storing data, or may be operatively connected to receive or transmit data, or both, to or from the one or more mass storage devices. However, a computer need not have these devices. Furthermore, a computer may be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio player, or a global positioning system (GPS) receiver, to name a few. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, including, for example, semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices); magnetic disks (e.g., internal hard disks and removable disks); magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0188] To provide for interaction with a user, implementations may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user, and a keyboard and pointing device (e.g., a mouse or trackball) with which the user can provide input to the computer. Other types of devices may also be used to provide for interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any suitable form, including sound, voice, or tactile input.

[0189] The implementation can be implemented in a computing system that includes a back-end component, such as a data server, or includes a middleware component (e.g., an application server), or includes a front-end component (e.g., a client computer with a graphical user interface or a web browser through which a user can interact with the implementation), or includes any suitable combination of one or more such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any suitable form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks ("LANs") and wide area networks ("WANs"), such as the Internet.

[0190] A computing system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0191] Although this specification contains many details, these should not be interpreted as limitations on the scope of the present disclosure or the scope of what may be claimed, but rather as descriptions of features that are specific to a particular implementation. Specific features described in this specification in the context of separate implementations may also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation may also be implemented in multiple implementations individually or in any suitable subcombination. Furthermore, although features may be described above as functioning in a particular combination and even as initially protected, in some examples, one or more features from that combination may be implemented from the claimed combination, and the claimed combination may be directed to subcombinations or variations of subcombinations.

[0192] Likewise, although the operations are shown in the accompanying drawings in a particular order, this should not be construed as requiring that the operations be performed in the particular order shown, or that all illustrated operations be performed sequentially, in order to achieve the desired results. In certain circumstances, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system components in the implementations described above should not be construed as requiring such separation in all implementations, and it should be understood that the program components and systems described can generally be integrated together into a single software product or packaged into multiple software products.

[0193] Many implementations have been described. However, it should be understood that various modifications may be made without departing from the spirit and scope of the present invention. For example, various forms of the processes shown above may be employed by reordering steps, adding steps, or deleting steps. Therefore, other implementations are within the scope of the appended claims.

[0194] Various embodiments have been described in the description herein with reference to many specific details, which may vary from implementation to implementation. Therefore, the description and drawings should be regarded as illustrative, not restrictive. The sole and exclusive indication of the scope of the invention, and what the applicants expect to be the scope of the invention, is the literal and equivalent scope of the claims issued from this application in the specific form in which the claims are issued, including any subsequent amendments. Any definition of terms explicitly set forth herein for being included in such claims should be based on the meaning of such terms as used in the claims. In addition, when the term "also includes" is used in the previous description or the appended claims, the phrase may be followed by additional steps or entities, or sub-steps / sub-entities of the previously described steps or entities.

Claims

1. A method for simulating an autonomous vehicle, comprising: receiving, with one or more processors, data collected by a computing device of an autonomous vehicle during operation of the autonomous vehicle; generating, utilizing the one or more processors and based on the received data, a simulation of operation of the autonomous vehicle; identifying, with the one or more processors, at least a portion of the simulation that indicates a deviation between the collected data and a simulated operation of the autonomous vehicle; analyzing, with the one or more processors, the at least a portion of the simulation to generate a metric for the at least a portion of the simulation; as well as The metric is sent, using the one or more processors, to the computing device, wherein the computing device is configured to use the metric to avoid another deviation between the collected data and the simulated operation of the autonomous vehicle.

2. The method for simulation of an autonomous vehicle according to claim 1, wherein: A discrepancy between the collected data and the simulated operation occurs when the collected data indicates that the computing device indicated an alert of a potential collision and the simulated operation of the autonomous vehicle indicates that the alert is false.

3. The method for simulation of an autonomous vehicle according to claim 2, wherein: The alert includes a notification indicating an unexpected occurrence in the vicinity of the autonomous vehicle.

4. The method for simulation of an autonomous vehicle according to claim 3, wherein: The unexpected occurrence includes one or more of the following: unexpected blockage of the path through which the autonomous vehicle is traversing; unexpected presence of one or more objects or one or more creatures in the path of the autonomous vehicle; and unexpected weather conditions experienced during operation of the autonomous vehicle.

5. The method for simulation of an autonomous vehicle according to any one of claims 2 to 4, wherein: The alert includes a notification indicating that the computing device has failed.

6. The method for simulation of an autonomous vehicle according to claim 1 or 2, wherein: A discrepancy between the collected data and the simulated operation occurs when the collected data indicates that the computing device failed to indicate an alert of a potential collision, but the simulated operation of the autonomous vehicle indicates a potential collision.

7. The method for simulation of an autonomous vehicle according to claim 6, wherein: The alert includes a notification indicating an unexpected occurrence in the vicinity of the autonomous vehicle.

8. The method for simulation of an autonomous vehicle according to claim 7, wherein: The unexpected occurrence includes one or more of the following: unexpected blockage of the path through which the autonomous vehicle is traversing; unexpected presence of one or more objects or one or more creatures in the path of the autonomous vehicle; and unexpected weather conditions experienced during operation of the autonomous vehicle.

9. The method for simulation of an autonomous vehicle according to claim 6, wherein: The alert includes a notification indicating that the computing device has failed.

10. The method for simulation of an autonomous vehicle according to any one of claims 1 to 4, further comprising: storing the data collected by the computing device in a database, The receiving the collected data by using the one or more processors includes: receiving the collected data from the database.

11. The method for simulation of an autonomous vehicle according to any one of claims 1 to 4, wherein: The one or more processors implement a user interface that allows a user to select the autonomous vehicle from a plurality of autonomous vehicles displayed on the user interface, the user selection being performed prior to receiving the collected data using the one or more processors.

12. A system for simulation of an autonomous vehicle, comprising: a first database for storing data indicative of the operation of at least one module within a computing device of an autonomous vehicle; A simulator comprising one or more processors, wherein the one or more processors are configured to: receiving stored data from said first database, generating a simulation of the operation of the at least one module based on the received data, identifying at least a portion of the simulation that indicates a deviation between the collected data and a simulated operation of the autonomous vehicle, and analyzing the at least a portion of the simulation to generate a metric for the at least a portion of the simulation, wherein the metric is usable to avoid another deviation between the collected data and the simulated operation of the autonomous vehicle; as well as A second database is configured to store the metrics.

13. The system for simulation of an autonomous vehicle according to claim 12, wherein: The computing device is configured to use the metrics stored in the second database to avoid another deviation between the collected data and the simulated operation of the autonomous vehicle.

14. A system for simulation of an autonomous vehicle according to claim 12 or 13, wherein: The at least one module includes an autonomous vehicle emergency brake module, or AVEB module, configured to determine whether to activate brakes of the autonomous vehicle at a particular moment during operation of the autonomous vehicle.

15. The system for simulation of an autonomous vehicle according to claim 14, wherein: The metrics stored in the second database are used to generate an updated determination regarding whether to activate the braking of the autonomous vehicle at the particular moment during operation of the autonomous vehicle, wherein the updated determination is used to avoid the further deviation between the collected data and the simulated operation of the autonomous vehicle.

16. The system for simulation of an autonomous vehicle according to claim 12 or 13, wherein: The computing device of the autonomous vehicle further includes a planning module and a control module, The simulation of the operation of the at least one module includes simulation of the planning module and the control module, and The metrics stored in the second database instruct the planning module and the control module to act to avoid the other deviation between the collected data and the simulated operation of the autonomous vehicle.

17. A non-transitory computer-readable medium storing instructions, which, when executed by at least one programmable processor, cause the at least one programmable processor to perform operations comprising: receiving data collected by a computing device of an autonomous vehicle during operation of the autonomous vehicle; generating a simulation of operation of the autonomous vehicle based on the received data; identifying at least a portion of the simulation that indicates a deviation between the collected data and a simulated operation of the autonomous vehicle; analyzing the at least a portion of the simulation to generate a metric for the at least a portion of the simulation; as well as The metric is sent to the computing device, wherein the computing device is configured to use the metric to avoid further deviation between the collected data and the simulated operation of the autonomous vehicle.

18. A method for simulation of an autonomous vehicle, comprising: receiving, with one or more processors, data collected by a computing device of an autonomous vehicle during operation of the autonomous vehicle; generating, utilizing the one or more processors and based on the received data, a simulation of operation of the autonomous vehicle; identifying, with the one or more processors, at least a portion of the simulation that indicates a takeover from an autonomous driving mode in response to the computing device generating an alert indicating a potential collision; analyzing, with the one or more processors, the at least a portion of the simulation to determine a metric indicating whether a collision would have occurred if the takeover had not occurred; as well as The metric is sent, using the one or more processors, to the computing device, wherein the computing device is configured to use the metric to improve the accuracy of a future alert indicating another potential collision.

19. The method for simulation of an autonomous vehicle according to claim 18, wherein: The driver's takeover includes: the safety driver pressing the emergency brake.

20. The method for simulation of an autonomous vehicle according to claim 18 or 19, wherein: The taking over by the driver includes navigating the autonomous vehicle to a safe location away from the accident in the vicinity of the autonomous vehicle.

21. The method for simulation of an autonomous vehicle according to claim 20, wherein: The unexpected occurrence includes one or more of the following: unexpected blockage of the path through which the autonomous vehicle is traversing; unexpected presence of one or more objects or one or more creatures in the path of the autonomous vehicle; and unexpected weather conditions experienced during operation of the autonomous vehicle.

22. The method for simulation of an autonomous vehicle according to claim 18 or 19, wherein: The takeover is performed by the driver of the autonomous vehicle.

23. The method for simulation of an autonomous vehicle according to claim 18 or 19, wherein: The takeover is performed by a computer that is coupled to the computing device via a communications network and is configured to remotely control the operation of the autonomous vehicle.

24. The method for simulation of an autonomous vehicle according to claim 18 or 19, wherein: The computing device generates the alert in response to detecting a change in a vital sign of a driver of the autonomous vehicle exceeding a threshold, wherein the takeover is performed by a computer coupled to the computing device via a communications network and configured to remotely control the operation of the autonomous vehicle.

25. The method for simulation of an autonomous vehicle according to claim 18 or 19, further comprising: storing the data collected by the computing device in a database, The receiving the collected data by using the one or more processors includes: receiving the collected data from the database.

26. The method for simulation of an autonomous vehicle according to claim 18 or 19, wherein: The one or more processors implement a user interface that allows a user to select the autonomous vehicle from a plurality of autonomous vehicles, the user selection being performed prior to receiving the collected data using the one or more processors.

27. A system for simulation of an autonomous vehicle, comprising: a first database configured to store data received from a computing device of an autonomous vehicle, wherein the stored data is indicative of an operation of the autonomous vehicle; A simulator comprising one or more processors, wherein the one or more processors are configured to: receiving stored data from said first database, generating a simulation of the operation of the autonomous vehicle based on the received data, identifying at least a portion of the simulation that indicates a takeover from the autonomous driving mode in response to the computing device generating an alert indicating a potential collision, and analyzing the at least a portion of the simulation to determine a metric indicating whether a collision would have occurred if the takeover had not occurred; as well as A second database is configured to store the metrics.

28. The system for simulation of an autonomous vehicle according to claim 27, wherein: The computing device is configured to use the metrics stored in the second database to improve the accuracy of future alerts indicating another potential collision.

29. A system for simulation of an autonomous vehicle according to claim 27 or 28, wherein: The computing device of the autonomous vehicle includes a planning module and a control module, The simulation of the operation of the autonomous vehicle includes simulation of the planning module and the control module, and The metrics stored in the second database instruct the planning module and the control module to act to improve the accuracy of future alerts indicating another potential collision.

30. The system for simulation of an autonomous vehicle according to claim 29, wherein: The planning module and the control module are each a software module implemented on a corresponding processor.

31. A non-transitory computer-readable medium storing instructions that, when executed by at least one programmable processor, cause the at least one programmable processor to perform operations comprising: receiving data collected by a computing device of an autonomous vehicle during operation of the autonomous vehicle; generating a simulation of operation of the autonomous vehicle based on the received data; identifying at least a portion of the simulation that indicates a takeover from the autonomous driving mode in response to the computing device generating an alert indicating a potential collision; analyzing the at least a portion of the simulation to determine a metric indicating whether a collision would have occurred if the takeover had not occurred; as well as The metric is sent to the computing device, where the computing device is configured to use the metric to improve the accuracy of a future alert indicating another potential collision.

32. A computer program product comprising a computer program for implementing the method according to any one of claims 1 to 11, 18 to 26 when the computer program is executed by a processor.

Images