Method and vehicle system for a vehicle

Abstract

The invention relates to a method for a vehicle and a vehicle system. Techniques for informing and explaining actions taken by an autonomous vehicle are described, including but not limited to: receiving a planned path of the vehicle, a state of the vehicle, and environmental data of an environment in which the vehicle is operating; receiving a deviation signal; determining whether the deviation signal was reported by a first system or a second system of the vehicle; in response, selecting a first set of simulators or a second set of simulators for simulating the vehicle in the environment; using the selected first set of simulators or the second set of simulators to simulate the vehicle in the environment; generating a message based on a result of the simulation; and presenting the message to at least one occupant of the vehicle.

Description

TECHNICAL FIELD

[0001] This specification relates to systems, methods, and computer program products for informing and explaining actions taken by an autonomous vehicle to individuals in the vicinity of the vehicle. BACKGROUND

[0002] An autonomous vehicle (AV) can use path planning to plan a route from a start location to an end location. A route is specified, and the vehicle proceeds along the route. The AV can temporarily or suddenly deviate from the route. For example, the AV can suddenly deviate from the planned path to avoid an imminent collision or to maintain a predefined lateral spacing from a nearby object. Such unexpected deviations can cause confusion among passengers or to remote operators of a command center. SUMMARY

[0003] A method for a vehicle, comprising: receiving, with at least one processor, a planned path of the vehicle, a state of the vehicle, and environmental data of an environment in which the vehicle is operating; receiving, with the at least one processor, a deviation signal; determining, with the at least one processor, whether the deviation signal is reported by a first system or a second system of the vehicle; in accordance with the deviation signal being reported by the first system of the vehicle, selecting, with the at least one processor, a first set of simulators for simulating the vehicle in the environment; in accordance with the deviation signal being reported by the second system of the vehicle, selecting, with the at least one processor, a second set of simulators for simulating the vehicle in the environment; simulating, with the at least one processor, the vehicle in the environment using the selected first or second set of simulators, the simulation including simulating the vehicle and at least one static or dynamic object; generating, with the at least one processor, a message based on a result of the simulation; and presenting, with the at least one processor, the message to at least one occupant of the vehicle.

[0004] A method for a vehicle, comprising: receiving, using at least one processor, data indicative of a planned path of the vehicle, a current state of the vehicle, and an environment in which the vehicle is operating; receiving, using the at least one processor, a first deviation signal or a second deviation signal; in accordance with receiving the first deviation signal, simulating, using a first simulator that takes the data as input, motion of the vehicle and an object to determine whether the object caused the vehicle to deviate from the planned path; in accordance with receiving the second deviation signal, simulating, using a second simulator that takes the data as input, motion of the vehicle and an object to determine whether the object caused the vehicle to deviate from the planned path; and in accordance with determining that the object caused the deviation, presenting, using the at least one processor, a message to an occupant in the vehicle, the message indicating the deviation from the planned path and the object identified as a cause of the deviation.

[0005] A vehicle system, comprising: at least one processor; and memory storing instructions that, when executed by the at least one processor, cause the at least one processor to perform the above method. BRIEF DESCRIPTION OF DRAWINGS

[0006] Figure 1 An example of an autonomous vehicle (AV) with autonomous capabilities is shown.

[0007] Figure 2 An example "cloud" computing environment is shown.

[0008] Figure 3 A computer system is shown.

[0009] Figure 4 An example architecture of an AV is shown.

[0010] Figure 5 An example of inputs and outputs that a perception system can use is shown.

[0011] Figure 6 A block diagram showing relationships between inputs and outputs of a planning system is shown.

[0012] Figure 7 A directed graph used in path planning is shown.

[0013] Figure 8 A block diagram showing inputs and outputs of a control system is shown.

[0014] Figure 9 A block diagram showing inputs, outputs, and components of a controller is shown.

[0015] Figure 10A block diagram of an example autonomous vehicle notification and post-action interpretation system is shown.

[0016] Figure 11 A block diagram showing the inputs, outputs, and components of an example simulation system used in an autonomous vehicle notification and action post-interpretation system.

[0017] Figure 12 A flowchart is shown for the process of notifying and explaining the actions of the vehicle. Detailed Implementation

[0018] In the following description, numerous specific details are set forth for purposes of explanation in order to provide a thorough understanding of the invention. However, it will be apparent that the invention may be practiced without these specific details. In other instances, well-known constructions and apparatuses are shown in block diagram form to avoid unnecessarily obscuring the invention.

[0019] In the accompanying drawings, for ease of description, a specific arrangement or order of schematic elements (such as those representing devices, modules, systems, instruction blocks, and data elements) is shown. However, those skilled in the art will understand that the specific order or arrangement of the schematic elements in the drawings is not intended to imply a requirement for a particular processing order or sequence, or a separation of processing procedures. Furthermore, the inclusion of schematic elements in the drawings is not intended to imply that such elements are required in all embodiments, nor is it intended to imply that features represented by such elements cannot be included in some embodiments or cannot be combined with other elements in some embodiments.

[0020] Furthermore, in the accompanying drawings, connecting elements, such as solid or dashed lines or arrows, are used to illustrate connections, relationships, or associations between two or more other schematic elements. The absence of any such connecting element does not imply that connections, relationships, or associations cannot exist. In other words, some connections, relationships, or associations between elements are not shown in the drawings so as not to obscure the content of this disclosure. Additionally, for ease of illustration, a single connecting element is used to represent multiple connections, relationships, or associations between elements. For example, if a connecting element represents communication of signals, data, or instructions, those skilled in the art will understand that such an element represents one or more signal paths (e.g., a bus) that may be necessary to affect the communication.

[0021] Reference will now be made in detail to embodiments, examples of which are illustrated in the accompanying drawings. Numerous specific details are set forth in the following detailed description in order to provide a thorough understanding of the various embodiments described. However, it will be apparent to those skilled in the art that the various embodiments described can be practiced without these specific details. In other instances, well-known methods, procedures, components, circuits, and networks have not been described in detail so as not to unnecessarily obscure aspects of the embodiments.

[0022] Each of the several features described below can be used independently of each other, or with any combination of other features. However, any individual feature can not solve any of the problems discussed above, or can only solve one of the problems discussed above. Some of the problems discussed above can not be fully addressed by any one feature described herein. Although a heading is provided, information related to a particular heading can also be found elsewhere in the specification. Embodiments are described herein according to the following outline:

[0023] 1. OVERALL SUMMARY

[0024] 2. SYSTEM SUMMARY

[0025] 3. AUTONOMOUS VEHICLE ARCHITECTURE

[0026] 4. AUTONOMOUS VEHICLE INPUT

[0027] 5. AUTONOMOUS VEHICLE PLANNING

[0028] 6. AUTONOMOUS VEHICLE CONTROL

[0029] 7. AUTONOMOUS VEHICLE NOTIFICATION AND POST-ACTION EXPLANATION SYSTEM

[0030] OVERALL SUMMARY

[0031] An autonomous vehicle (AV) can make a maneuver to avoid an impending collision or to maintain a predefined distance from a nearby object. In an example, the AV makes a maneuver to avoid an impending collision with an object. In another example, the AV makes a maneuver to maintain a predefined distance from other vehicles or objects that are getting closer to the AV or the AV’s planned path. Such a maneuver causes the AV to deviate from the planned path. This deviation can be unexpected or sudden (e.g., a sudden change in speed or acceleration) to the AV user (e.g., a passenger in the vehicle or an operator in a remote command center). Repeated deviations without proper notification or explanation can lead to a loss of confidence in the AV system by the user. The object(s) that caused such an event can be identified, and an explanation of how the object(s) caused the deviation will be reported to the user through visual and / or audio messages. An explanation system is implemented to provide the user with an explanation of the deviation from the planned path. A simulation can be made to determine the cause(s) of the vehicle’s deviation. For example, a collision avoidance signal is generated when the vehicle slows down to avoid a collision with an animal crossing the road. In this example, the collision avoidance signal prompts a corresponding simulation system to determine the cause of the slowdown. An explanation of the vehicle’s deviation (e.g., the vehicle slowed down to avoid a collision with an animal crossing the road) is broadcast to the vehicle’s passengers using audio and / or visual interfaces in the vehicle.

[0032] Some advantages of these techniques include: building user trust and confidence in the AV vehicle through an explanation of the autonomous vehicle's deviation from the planned path. The explanation occurs after the AV deviates, so it does not require user intervention. The explanation system uses data from the AV stack (e.g., planning or control circuitry), but does not consume the AV stack's computational resources, so it will not affect the AV stack's processing speed. The explanation system can alert a teleoperator (if one is present or available) to intervene if necessary. A backup simulator ensures system operation in the event of a primary simulator failure. A history of deviation events and their respective explanations is saved in a log or database to allow future review or diagnosis.

[0033] System Overview

[0034] Figure 1 An example of an AV 100 with autonomous capabilities is shown.

[0035] As used herein, the term "autonomous capabilities" refers to a function, feature, or facility that enables a vehicle to operate, in part or in whole, without real-time human intervention, including but not limited to fully AV, highly AV, and conditional AV.

[0036] As used herein, an autonomous vehicle (AV) is a vehicle with autonomous capabilities.

[0037] As used herein, a "vehicle" includes a means of transporting goods or people. For example, a car, bus, train, airplane, drone, truck, boat, ship, submarine, spacecraft, etc. A self-driving car is an example of a vehicle.

[0038] As used herein, a "trajectory" refers to a path or route that navigates an AV from a first spatiotemporal location to a second spatiotemporal location. In embodiments, the first spatiotemporal location is referred to as an initial or starting location, and the second spatiotemporal location is referred to as a destination, final location, target, target location, or target location. In some examples, a trajectory is composed of one or more segments (e.g., sections of a road), and each segment is composed of one or more blocks (e.g., a portion of a lane or intersection). In embodiments, the spatiotemporal locations correspond to real-world locations. For example, a spatiotemporal location is a pickup or drop-off location for a person or cargo to board or disembark.

[0039] As used herein, a “sensor(s)” includes one or more hardware components for detecting information related to the sensor’s surrounding environment. Some hardware components can include sensing components (e.g., image sensors, biometric sensors), transmission and / or reception components (e.g., laser or radio frequency wave emitters and receivers), electronic components such as analog-to-digital converters, data storage devices such as RAM and / or non-volatile memory, software or firmware components, and data processing components such as application-specific integrated circuits, microprocessors, and / or microcontrollers.

[0040] As used herein, a “scene description” is a data structure (e.g., a list) or data stream that includes one or more classified or labeled objects detected by one or more sensors on an AV or one or more classified or labeled objects provided by a source external to the AV.

[0041] As used herein, a “roadway” is a physical area that can be traversed by a vehicle, and can correspond to a named thoroughfare (e.g., a city street, an interstate highway, etc.) or can correspond to an unnamed thoroughfare (e.g., a driveway within a house or office building, a section of a parking lot, a section of an empty parking lot, a dirt path in a rural area, etc.). Because some vehicles (e.g., four-wheel drive pickup trucks, sport utility vehicles (SUVs), etc.) are capable of traversing a variety of physical areas that are not particularly well-suited for vehicle travel, a “roadway” can be any physical area that has not been formally defined as a thoroughfare by a municipality or other governmental or administrative body.

[0042] As used herein, a "lane" is a portion of a roadway that can be traversed by a vehicle. Sometimes a lane is identified based on lane markings. For example, a lane can correspond to most or all of the space between lane markings, or only a portion of the space (e.g., less than 50%) between lane markings. For example, a roadway with lane markings far apart can accommodate two or more vehicles such that one vehicle can pass another without crossing a lane marking, and thus can be interpreted as a lane being narrower than the space between lane markings, or as two lanes between lanes. Lanes can also be interpreted in the absence of lane markings. For example, a lane can be defined based on physical features of the environment (e.g., rocks and trees along a road in a rural area, or natural obstacles that should be avoided such as in underdeveloped areas). Lanes can also be interpreted independent of lane markings or physical features. For example, a lane can be interpreted based on an arbitrary path in an area that lacks obstacles that would otherwise lack features that would be interpreted as lane boundaries. In an example scenario, an AV can interpret a lane through an unobstructed portion of a field or open space. In another example scenario, an AV can interpret a lane through a wide (e.g., wide enough for two or more lanes) roadway without lane markings. In this scenario, the AV can communicate information about the lane to other AVs so that the other AVs can use the same lane information to coordinate path planning between AVs.

[0043] The term "over-the-air (OTA) client" includes any AV, or any electronic device (e.g., computer, controller, IoT device, electronic control unit (ECU)) embedded in, coupled to, or in communication with an AV.

[0044] The term "over-the-air (OTA) update" means any update, change, deletion, or addition to software, firmware, data, or configuration settings, or any combination thereof, delivered to an OTA client using proprietary and / or standardized wireless communication technologies, including but not limited to: cellular mobile communication (e.g., 2G, 3G, 4G, 5G), wireless radio area networks (e.g., WiFi), and / or satellite internet.

[0045] The term "edge node" refers to one or more edge devices coupled to a network that provide a portal for communication with AVs and can communicate with other edge nodes and cloud-based computing platforms to schedule and deliver OTA updates to OTA clients.

[0046] The term "edge device" refers to a device that implements an edge node and provides a physical wireless access point (AP) to a corporate or service provider (e.g., VERIZON, AT&T) core network. Examples of edge devices include, but are not limited to: a computer, a controller, a transmitter, a router, a routing switch, an integrated access device (IAD), a multiplexer, a metropolitan area network (MAN) and a wide area network (WAN) access device.

[0047] "one or more" includes a function performed by one element, a function performed by more than one element, e.g., in a distributed manner, a function performed by one element and a function performed by more than one element, a function performed by one element and a function performed by several elements, or any combination of the above.

[0048] It will also be understood that, although the terms "first," "second," etc. can be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first contact could be termed a second contact, and, similarly, a second contact could be termed a first contact, without departing from the scope of the various described embodiments. The first contact and the second contact are both contacts, but they are not the same contact.

[0049] The terminology used in the description of the various embodiments described herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used in the description of the various embodiments and the appended claims, the singular forms "a," "an," and "the" are intended to include plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items. It will be further understood that the terms "comprises," "comprising," "includes," "including," "has," "having," "has" and / or "having," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0050] As used herein, the term "if' can be construed to mean "when" or "if when" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [a stated condition or event] is detected" can be construed to mean "upon determining" or "in response to determining" or "upon detecting [the stated condition or event]" or "in response to detecting [the stated condition or event]," depending on the context.

[0051] As used herein, an AV system refers to an array of AVs and the hardware, software, stored data, and real-time generated data that support the operation of the AVs. In embodiments, the AV system is incorporated within the AVs. In embodiments, the AV system is distributed across several locations. For example, some software of the AV system is in the cloud similar to the cloud computing environment 200 described below with respect to Figure 2 implemented on a cloud computing environment of the cloud computing environment 200 described below.

[0052] In general, this document describes technology applicable to any vehicle with one or more levels of autonomy, including fully AVs, highly AVs, and conditional AVs, such as so-called Level 5, Level 4, and Level 3 vehicles, respectively (see SAE International Standard J3016: Classification and Definition of Degrees of Automation for Motor Vehicles, incorporated by reference in its entirety into this document for more details on vehicle autonomy levels). The technology described in this document is also applicable to partial AVs and driver-assisted vehicles, such as so-called Level 2 and Level 1 vehicles (see SAE International Standard J3016: Classification and Definition of Degrees of Automation for Motor Vehicles). In embodiments, one or more Level 1, Level 2, Level 3, Level 4, and Level 5 vehicle systems can automatically perform certain vehicle operations (e.g., steering, braking, and use of maps) under certain operating conditions based on processing of sensor inputs. The technology described in this document can benefit vehicles at any level ranging from fully AVs to human-operated vehicles.

[0053] AVs have advantages over vehicles that require a human driver. One advantage is safety. For example, in 2016, the United States experienced 6 million motor vehicle crashes, 2.4 million people injured, 40,000 people killed, and 100 million vehicle crash incidents, with an estimated societal cost of $910 billion. From 1965 to 2015, the number of traffic fatalities per 100 million miles traveled in the United States has decreased from about 6 to about 1, in part due to additional safety measures deployed in vehicles. For example, warnings of an additional half-second associated with an impending collision are believed to mitigate 60% of rear-end collisions. However, passive safety features (e.g., seat belts, airbags) can have reached their limit in improving this number. Thus, active safety measures such as automated control of vehicles are a possible next step in improving these statistics. Since a human driver is believed to be the cause of a serious pre-crash event in 95% of crashes, automated driving systems can achieve better safety outcomes by, for example, reliably identifying and avoiding emergencies better than a human, making better decisions than a human, better complying with traffic laws than a human, and better predicting future events than a human, and reliably controlling the vehicle better than a human.

[0054] Reference Figure 1 The AV system 120 causes the vehicle 100 to operate along a trajectory 198 through the environment 190 to a destination 199 (sometimes referred to as a final location) while avoiding objects (e.g., natural obstacles 191, vehicles 193, pedestrians 192, cyclists, and other obstacles) and obeying road rules (e.g., operating rules or driving preferences).

[0055] In embodiments, the AV system 120 includes a device 101 for receiving and operating on operational commands from the computer processor 146. The term "operational command" is used to denote an executable instruction (or set of instructions) that causes the vehicle to take an action (e.g., a driving maneuver). Operational commands can include, without limitation, instructions for causing the vehicle to start moving forward, stop moving forward, start moving backward, stop moving backward, accelerate, decelerate, make a left turn, and make a right turn. In embodiments, the computer processor 146 is similar to the processor 304 described below with reference to FIG. 3. Examples of the device 101 include a steering controller 102, a brake 103, a gear, an accelerator pedal or other acceleration control mechanism, a windshield wiper, a side door lock, a window control, and a turn indicator. Figure 3

[0056] In embodiments, the AV system 120 includes sensors 121 for measuring or inferring properties of the state or condition of the vehicle 100, such as the AV's position, linear and angular velocity and linear and angular acceleration, and heading (e.g., the direction of the front end of the vehicle 100). Examples of the sensors 121 are a GPS, an inertial measurement unit (IMU) that measures both linear acceleration and angular rate of the vehicle, a wheel rate sensor for measuring or estimating wheel slip, a wheel brake pressure or brake torque sensor, an engine torque or wheel torque sensor, and a steering angle and angular rate sensor.

[0057] In embodiments, the sensors 121 also include sensors for sensing or measuring properties of the AV's environment. For example, monocular or stereo video cameras 122 in the visible, infrared, or thermal (or both) light spectrum, LiDAR 123, RADAR, ultrasonic sensors, time-of-flight (TOF) depth sensors, speed sensors, temperature sensors, humidity sensors, and precipitation sensors.

[0058] In embodiments, the AV system 120 includes a data storage unit 142 and a memory 144 for storing machine instructions associated with the computer processor 146 or data collected by the sensors 121. In embodiments, the data storage unit 142 is similar to the data storage unit 342 described below with reference to FIG. 3. Figure 3 ​The described ROM 308 or storage 310 are similar. In embodiments, the memory 144 is similar to the main memory 306 described below. In embodiments, the data storage unit 142 and the memory 144 store historical, real-time, and / or predictive information about the environment 190. In embodiments, the stored information includes maps, driving performance, traffic congestion updates, or weather conditions. In embodiments, data about the environment 190 is transmitted from the remote database 134 to the vehicle 100 through a communication channel.

[0059] In embodiments, the AV system 120 includes communication devices 140 for transmitting properties of other vehicles' states and conditions, such as position, linear and angular velocity, linear and angular acceleration, and linear and angular heading, measured or inferred to the vehicle 100. These devices include vehicle-to-vehicle (V2V) and vehicle-to-infrastructure (V2I) communication devices and devices for wireless communication through point-to-point or ad hoc networks or both. In embodiments, the communication devices 140 communicate across the electromagnetic spectrum, including radio and optical communications, or other media (e.g., air and acoustic media). The combination of vehicle-to-vehicle (V2V) and vehicle-to-infrastructure (V2I) communication (and, in some embodiments, one or more other types of communication) is sometimes referred to as vehicle-to-everything (V2X) communication. V2X communication is generally in compliance with one or more communication standards for communication with and between AVs.

[0060] In embodiments, the communication devices 140 include a communication interface. For example, a wired, wireless, WiMAX, Wi-Fi, Bluetooth, satellite, cellular, optical, near field, infrared, or radio interface. The communication interface transmits data from the remote database 134 to the AV system 120. In embodiments, the remote database 134 is embedded in a cloud computing environment 200 as described in Figure 2 In embodiments, the communication devices 140 transmit data collected from the sensors 121 or other data related to the operation of the vehicle 100 to the remote database 134. In embodiments, the communication devices 140 transmit information related to teleoperation to the vehicle 100. In some embodiments, the vehicle 100 communicates with other remote (e.g., "cloud") servers 136.

[0061] In embodiments, the remote database 134 also stores and transmits digital data (e.g., stores data such as road and street locations). This data is stored in the memory 144 on the vehicle 100 or transmitted from the remote database 134 to the vehicle 100 through a communication channel.

[0062] In embodiments, the remote database 134 stores and transmits historical information (e.g., speed and acceleration profiles) related to driving attributes of vehicles that have previously traveled along the trajectory 198 at similar times of day. In one implementation, such data can be stored in the memory 144 on the vehicle 100, or transmitted from the remote database 134 to the vehicle 100 over a communication channel.

[0063] The computer processor 146 located on the vehicle 100 generates control actions algorithmically based on both real-time sensor data and a priori information, allowing the AV system 120 to perform its autonomous driving capabilities.

[0064] In embodiments, the AV system 120 includes computer peripherals 132 coupled to the computer processor 146 for providing information and reminders to a user (e.g., a passenger or a remote user) of the vehicle 100 and receiving input from the user. In embodiments, the peripherals 132 are similar to the display 312, input device 314, and cursor controller 316 discussed below with reference to FIG. 3. The coupling is wireless or wired. Any two or more of the interface devices can be integrated into a single device. Figure 3

[0065] In embodiments, the AV system 120 receives and enforces a privacy level of a passenger, for example, specified by the passenger or stored in a profile associated with the passenger. The privacy level of the passenger determines how the use of certain information associated with the passenger (e.g., passenger comfort data, biometric data, etc.) stored in the passenger profile and / or stored on the cloud server 136 and associated with the passenger profile is permitted. In embodiments, the privacy level specifies certain information associated with the passenger that is deleted upon completion of a ride. In embodiments, the privacy level specifies certain information associated with the passenger and identifies one or more entities that are authorized to access the information. Examples of the specified entities that are authorized to access the information can include other AVs, third-party AV systems, or any entity that can potentially have access to the information.

[0066] The privacy level of the passenger can be specified at one or more levels of granularity. In embodiments, the privacy level identifies specific information to be stored or shared. In embodiments, the privacy level applies to all information associated with the passenger, such that the passenger can specify that her personal information is not stored or shared. The specification of entities that are permitted to access specific information can also be specified at various levels of granularity. Various sets of entities that are permitted to access specific information can include, for example, other AVs, the cloud server 136, specific third-party AV systems, etc.

[0067] ​In embodiments, the AV system 120 or the cloud server 136 determines whether the AV 100 or another entity has access to certain information associated with a passenger. For example, a third party AV system attempting to access passenger input related to a particular spatiotemporal location must obtain authorization, e.g., from the AV system 120 or the cloud server 136, to access information associated with the passenger. For example, the AV system 120 uses the passenger's specified privacy level to determine whether the passenger input related to the spatiotemporal location can be presented to the third party AV system, the AV 100, or another AV. This enables the passenger's privacy level to specify which other entities are allowed to receive data related to the passenger's actions or other data associated with the passenger.

[0068] Figure 2 An example "cloud" computing environment is shown. Cloud computing is a model of service delivery for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g. networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal management effort or interaction with a provider of the service. This cloud model can be comprised of various characteristics such as on-demand self-service, broad network access, resource pooling, rapid elasticity, measured service, and software as a service. A cloud computing environment is service oriented, allowing a provider to capture pervasive Figure 2 The cloud computing environment 200 includes cloud data centers 204a, 204b, and 204c interconnected by a cloud 202. The data centers 204a, 204b, and 204c provide cloud computing services for computer systems 206a, 206b, 206c, 206d, 206e, and 206f connected to the cloud 202.

[0069] The cloud computing environment 200 includes one or more cloud data centers. Generally, a cloud data center (e.g., the cloud data center 204a shown in Figure 2 refers to the physical arrangement of servers that make up a cloud (e.g., the cloud 202 shown in Figure 2 For example, servers are physically arranged in rooms, groups, rows, and racks in a cloud data center. A cloud data center has one or more areas that include one or more server rooms. Each room has one or more rows of servers, and each row includes one or more racks. Each rack includes one or more individual server nodes. In some implementations, servers in an area, room, rack, and / or row are arranged into groups based on physical infrastructure requirements of the data center facility, including power, energy, heat, heat sources, and / or other requirements. In embodiments, a server node is similar to the computer system described in Figure 3 The data center 204a has many computer systems distributed across multiple racks.

[0070] The cloud 202 includes cloud data centers 204a, 204b, and 204c and networks and network resources (e.g., network equipment, nodes, routers, switches, and network cables) used to connect and facilitate access by the computing systems 206a-f to cloud computing services. In embodiments, the network represents any combination of one or more local networks, wide-area networks, or internetworks coupled by the use of terrestrial or satellite link deployments of wired or wireless links. Data exchanged over the network is transmitted using a variety of network layer protocols, such as Internet Protocol (IP), Multiprotocol Label Switching (MPLS), Asynchronous Transfer Mode (ATM), Frame Relay, and the like. Moreover, in embodiments where the network represents a combination of multiple sub-networks, different network layer protocols are used on each underlying sub-network. In some embodiments, the network represents one or more internetworks, such as the public Internet, and the like.

[0071] The computing systems 206a-f or cloud computing service consumers are connected to the cloud 202 through network links and network adapters. In embodiments, the computing systems 206a-f are implemented as various computing devices, such as servers, desktops, laptops, tablets, smartphones, Internet of Things (IoT) devices, AVs (including cars, drones, spacecraft, trains, buses, and the like), and consumer electronics. In embodiments, the computing systems 206a-f are implemented in or as part of other systems.

[0072] Figure 3 A computer system 300 is shown. In implementations, the computer system 300 is a special-purpose computing device. The special-purpose computing device is hard-wired to perform the techniques, or includes digital electronic devices such as one or more application-specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs) that are persistently programmed to perform the techniques, or includes one or more general purpose hardware processors programmed to perform the techniques pursuant to program instructions in firmware, memory, other storage, or a combination. Such special-purpose computing devices can also combine custom hard-wired logic, ASICs, or FPGAs with custom programming to accomplish the techniques. In various embodiments, the special-purpose computing device is a

[0073] In embodiments, the computer system 300 includes a bus 302 or other communication mechanism for communicating information, and a processor 304 coupled with bus 302 for processing information. The processor 304 is, for example, a general-purpose microprocessor. The computer system 300 also includes a main memory 306, such as a random access memory (RAM) or other dynamic storage device, coupled to bus 302 for storing information and instructions to be executed by processor 304. In one implementation, the main memory 306 is used for storing temporary variables or other intermediate information during execution of instructions to be executed by processor 304. Such instructions can be stored or implemented in non-transitory storage media accessible to processor 304, such as storage media 310, when such instructions are available to the processor 304. The computer system 300 is, therefore, a special purpose machine that operates to perform the operations specified in the instructions.

[0074] In embodiments, the computer system 300 also includes a read only memory (ROM) 308 or other static storage device coupled to bus 302 for storing static information and instructions for processor 304. A storage device 310, such as a magnetic disk, optical disk, solid-state drive, or three-dimensional cross-point memory, is provided and coupled to bus 302 for storing information and instructions.

[0075] In embodiments, the computer system 300 is coupled via bus 302 to a display 312, such as a cathode ray tube (CRT), liquid crystal display (LCD), plasma display, light emitting diode (LED) display, or organic light emitting diode (OLED) display, for displaying information to a computer user. An input device 314, including alphanumeric and other keys, is coupled to bus 302 for communicating information and command selections to processor 304. Another type of user input device is cursor control 316, such as a mouse, a trackball, a touch display, or cursor direction keys for communicating direction information and command selections to processor 304 and for

[0076] According to one embodiment, the techniques herein are performed by computer system 300 in response to processor 304 executing one or more sequences of one or more instructions contained in main memory 306. Such instructions can be read into main memory 306 from another storage medium, such as storage device 310. Execution of the sequences of instructions contained in main memory 306 causes processor 304 to perform the process steps described herein. In alternative embodiments, hard-wired circuitry can be used in place of or in combination with software instructions.

[0077] The term "storage media" as used herein refers to any non-transitory media that store data and / or instructions that cause a machine to operate in a specific fashion. Such storage media include non-volatile media and / or volatile media. Non-volatile media include, for example, optical or magnetic disks, such as storage device 310. Volatile media include dynamic memory, such as main memory 306. Common forms of storage media include, for example, a floppy disk, a flexible disk, hard disk, solid-state drive, magnetic tape, or any other magnetic data storage medium, a CD-ROM, any other optical data storage medium, any physical medium with patterns of holes, a RAM, a PROM, and EPROM, a FLASH-EPROM, NVRAM, or any other memory chip or cartridge.

[0078] Storage media are distinct from, but can be used in combination with, transmission media. Transmission media participate in transferring information between storage media. For example, transmission media include coaxial cables, copper wire, and optical fibers, including the wires that comprise bus 302. Transmission media can also take the form of acoustic or light waves, such as those generated during radio frequency and infrared data communications.

[0079] In embodiments, various forms of media are involved in carrying one or more sequences of one or more instructions to processor 304 for execution. For example, the instructions are initially carried on a magnetic disk or solid-state drive of a remote computer. The remote computer loads the instructions into its dynamic memory and sends the instructions over a telephone line using a modem. A local modem within computer system 300 receives the data on the telephone line and uses an infrared transmitter to convert the data to an infrared signal. An infrared detector within computer system 300 receives the infrared signal and appropriate circuitry places the data in bus 302. Bus 302 carries the data to memory 306, from which processor 304 retrieves and executes the instructions. The instructions received by memory 306 can optionally be stored on storage device 310 either before or after execution by processor 304.

[0080] Computer system 300 also includes a communication interface 318 coupled to bus 302. Communication interface 318 provides a two-way data communication coupling to a network link 320 that is connected to a local network 322. For example, communication interface 318 is an integrated services digital network (ISDN) card, cable modem, satellite modem, or a modem to provide a data communication connection to a corresponding type of telephone line. As another example, communication interface 318 is a local area network (LAN) card to provide a data communication connection to a compatible LAN. Wireless links are also implemented in some implementations. In any such implementation, communication interface 318 sends and receives electrical, electromagnetic or optical signals that carry digital data streams representing various types of information.

[0081] Network link 320 typically provides data communication to other data devices via one or more networks. For example, network link 320 provides connectivity to host computer 324 or to a cloud data center or device operated by Internet Service Provider (ISP) 326 via local network 322. ISP 326, in turn, provides data communication services via a worldwide packet data communication network now commonly referred to as the “Internet” 328. Both local network 322 and Internet 328 use electrical, electromagnetic, or optical signals carrying digital data streams. Signals through various networks and signals on network link 320 via communication interface 318 are example forms of transmission media carrying digital data entering and leaving computer system 300. In embodiments, network 320 includes the above-described... Figure 2 The cloud 202 or a part of the cloud 202.

[0082] Computer system 300 sends messages and receives data including program code via (one or more) networks, network links 320, and communication interfaces 318. In an embodiment, computer system 300 receives code for processing. The received code is executed by processor 304 upon receipt and / or stored in storage device 310, or in other non-volatile storage devices for later execution.

[0083] Autonomous Vehicle Architecture

[0084] Figure 4 Showing for AV (e.g., Figure 1 The example architecture 400 of the vehicle 100 shown is illustrated. Architecture 400 includes a sensing system 402 (sometimes referred to as a sensing circuit), a planning system 404 (sometimes referred to as a planning circuit), a control system 406 (sometimes referred to as a control circuit), a positioning system 408 (sometimes referred to as a positioning circuit), and a database system 410 (sometimes referred to as a database circuit). Each system plays a role in the operation of the vehicle 100. Commonly, systems 402, 404, 406, 408, and 410 can be... Figure 1The illustrated portion of the AV system 120. In some embodiments, any of the systems 402, 404, 406, 408, and 410 are a combination of computer software (e.g., executable code stored on a computer-readable medium) and computer hardware (e.g., one or more microprocessors, microcontrollers, application-specific integrated circuits [ASICs], hardware memory devices, other types of integrated circuits, other types of computer hardware, or a combination of any or all of these). Each of the systems 402, 404, 406, 408, and 410 is sometimes referred to as processing circuitry (e.g., computer hardware, computer software, or a combination of both). A combination of any or all of the systems 402, 404, 406, 408, and 410 is also an example of processing circuitry.

[0085] In use, the planning system 404 receives data representing the destination 412 and determines data representing a trajectory 414 (sometimes referred to as a route) that the vehicle 100 can travel in order to reach (e.g., arrive at) the destination 412. To cause the planning system 404 to determine the data representing the trajectory 414, the planning system 404 receives data from the perception system 402, the localization system 408, and the database system 410.

[0086] The perception system 402 uses one or more sensors 121, e.g., also as illustrated in Figure 1 to identify nearby physical objects. The objects are classified (e.g., grouped into types such as pedestrians, bicycles, cars, traffic signs, etc.) and a scene description including the classified objects 416 is provided to the planning system 404.

[0087] The planning system 404 also receives data representing the AV position 418 from the localization system 408. The localization system 408 determines the AV position by using data from the sensors 121 and data (e.g., geographic data) from the database system 410 to compute a position. For example, the localization system 408 uses data from a GNSS (Global Navigation Satellite System) sensor and geographic data to compute the AV's longitude and latitude. In embodiments, the data used by the localization system 408 includes a high-precision map with lane geometry properties, a map describing road network connectivity properties, a map describing lane physical properties such as traffic speed, traffic volume, number of vehicle and bicycle lanes, lane width, lane traffic direction, or lane marking type and location, or a combination of these, and a map describing spatial locations of road features such as crosswalks, traffic signs, or other types of travel signals. In embodiments, the high-precision map is constructed by adding data via automatic or manual annotation to a low-precision map.

[0088] The control system 406 receives data representing trajectory 414 and data representing AV position 418, and operates the AV control functions 420a-420c (e.g., steering, throttle, braking, ignition) in a manner that will cause the vehicle 100 to travel along trajectory 414 to reach destination 412. For example, if trajectory 414 includes a left turn, the control system 406 will operate the control functions 420a-420c in such a way that the steering angle of the steering function will cause the vehicle 100 to turn left, and the throttle and brake will cause the vehicle 100 to stop and wait for passing pedestrians or vehicles before making the turn.

[0089] Autonomous vehicle input

[0090] Figure 5 The sensing system 402 is shown. Figure 4 The inputs used are 502a-502d (e.g., Figure 1 Examples of sensor 121 and outputs 504a-504d (e.g., sensor data) are shown. One input 502a is a LiDAR (light detection and ranging) system (e.g., Figure 1 The LiDAR system shown is 123. LiDAR is a technique that uses light (e.g., a beam of light such as infrared light) to obtain data related to physical objects in its line of sight. A LiDAR system produces LiDAR data as output 504a. For example, LiDAR data is used to construct... Figure 1 The environment shown is a collection of 3D or 2D points (also known as point clouds) representing the environment 190.

[0091] Another input 502b is the RADAR (radar) system. RADAR is a technology that uses radio waves to obtain data related to nearby physical objects. RADAR can obtain data related to objects that are not within the line of sight of a LiDAR system. The RADAR system produces RADAR data as output 504b. For example, RADAR data is used to construct... Figure 1 The environment 190 shown represents one or more radio frequency electromagnetic signals.

[0092] Another input 502c is a camera system. The camera system uses one or more cameras (e.g., digital cameras using light sensors such as charge-coupled devices [CCDs]) to acquire information about nearby physical objects. The camera system produces camera data as output 504c. The camera data is often in the form of image data (e.g., data in image data formats such as RAW, JPEG, PNG, etc.). In some examples, the camera system has multiple independent cameras, e.g., for the purpose of stereoscopic imagery (stereo vision), which enables the camera system to perceive depth. Although the objects perceived by the camera system are described here as being "nearby," this is relative to the AV. In some embodiments, the camera system is configured to "see" objects that are far away (e.g., up to 1 kilometer or more in front of the AV). Thus, in some embodiments, the camera system has features such as sensors and lenses that are optimized for perceiving objects that are far away.

[0093] Another input 502d is a traffic light detection (TLD) system. The TLD system uses one or more cameras to acquire information about traffic lights, street signs, and other physical objects that provide visual navigation information. The TLD system produces TLD data as output 504d. The TLD data is often in the form of image data (e.g., data in image data formats such as RAW, JPEG, PNG, etc.). The TLD system differs from a system that includes a camera in that the TLD system uses a camera with a wide field of view (e.g., using a wide-angle lens or a fisheye lens) to acquire information about as many physical objects that provide visual navigation information as possible, so that the vehicle 100 has access to all relevant navigation information provided by those objects. For example, the TLD system has a field of view of about 120 degrees or more.

[0094] In some embodiments, the outputs 504a-504d are combined using sensor fusion techniques. Thus, the individual outputs 504a-504d are provided to other systems of the vehicle 100 (e.g., to the planning system 404 as shown), or a combined output can be provided to the other systems in the form of a single combined output or multiple combined outputs of the same type (e.g., using the same combination technique or combining the same outputs, or both) or a single combined output or multiple combined outputs of different types (e.g., using different respective combination techniques or combining different respective outputs, or both). In some embodiments, early fusion techniques are used. Early fusion techniques are characterized by combining the outputs before applying one or more data processing steps to the combined output. In some embodiments, late fusion techniques are used. Late fusion techniques are characterized by combining the outputs after applying one or more data processing steps to the individual outputs. Figure 4 In some embodiments, the outputs 504a-504d are combined using sensor fusion techniques. Thus, the individual outputs 504a-504d are provided to other systems of the vehicle 100 (e.g., to the planning system 404 as shown), or a combined output can be provided to the other systems in the form of a single combined output or multiple combined outputs of the same type (e.g., using the same combination technique or combining the same outputs, or both) or a single combined output or multiple combined outputs of different types (e.g., using different respective combination techniques or combining different respective outputs, or both). In some embodiments, early fusion techniques are used. Early fusion techniques are characterized by combining the outputs before applying one or more data processing steps to the combined output. In some embodiments, late fusion techniques are used. Late fusion techniques are characterized by combining the outputs after applying one or more data processing steps to the individual outputs.

[0095] Path planning

[0096] Figure 6 Show (for example, as) Figure 4 The diagram 600 illustrates the relationship between the inputs and outputs of the planning system 404. Generally, the output of the planning system 404 is a route 602 from a starting point 604 (e.g., a source location or initial location) to an ending point 606 (e.g., a destination or final location). Route 602 is typically defined by one or more road segments. For example, a road segment refers to the distance traveled over at least a portion of a street, road, highway, driveway, or other physical area suitable for vehicle travel. In some examples, such as if the vehicle 100 is an off-road capable vehicle like a four-wheel drive (4WD) or all-wheel drive (AWD) car, SUV, or pickup truck, route 602 includes “off-road” segments such as unpaved paths or open fields.

[0097] In addition to route 602, the planning system also outputs lane-level route planning data 608. Lane-level route planning data 608 is used to navigate segments of route 602 at specific times based on conditions. For example, if route 602 comprises a multi-lane highway, lane-level route planning data 608 includes trajectory planning data 610, which vehicle 100 can use to select a lane from the multiple lanes based on factors such as whether an exit is nearby, whether another vehicle is present in one or more of the multiple lanes, or other factors that change over a period of minutes or less. Similarly, in some implementations, lane-level route planning data 608 includes a speed constraint 612 specific to a segment of route 602. For example, if the segment includes pedestrians or unexpected traffic, speed constraint 612 can limit vehicle 100 to a slower speed than expected, such as a speed limit based on the segment's speed limit data.

[0098] In an embodiment, the inputs to the planning system 404 include (e.g., from...) Figure 4 The database system 410 shown contains database data 614 and current location data 616 (e.g., Figure 4 The AV position shown is 418), (for example, for use with Figure 4 The destination data 618 and object data 620 shown for destination 412 (e.g., as shown) Figure 4The perception system 402 shown perceives classified objects 416. In some embodiments, database data 614 includes rules used during planning. The rules are specified using a formal language (e.g., Boolean logic). At least some of these rules will apply to any given situation encountered by vehicle 100. A rule applies to a given situation if it has conditions satisfied based on information available to vehicle 100 (e.g., information about the surrounding environment). Rules can have priorities. For example, a rule “move to the leftmost lane if the road is a highway” can have a lower priority than “move to the rightmost lane if the exit is within a mile.”

[0099] Figure 7 This is illustrated in path planning (e.g., by planning system 404). Figure 4 The directed graph used is 700. Generally speaking, such as... Figure 7 The directed graph 700 shown is used to determine any path between a starting point 702 and an ending point 704. In the real world, the distance separating the starting point 702 and the ending point 704 may be relatively large (e.g., in two different urban areas) or relatively small (e.g., two intersections in adjacent city blocks or two lanes of a multi-lane road).

[0100] In an embodiment, the directed graph 700 has nodes 706a-706d representing different locations that the vehicle 100 may occupy between the starting point 702 and the ending point 704. In some examples, for instance, when the starting point 702 and the ending point 704 represent different urban areas, nodes 706a-706d represent road segments. In some examples, for instance, when the starting point 702 and the ending point 704 represent different locations on the same road, nodes 706a-706d represent different locations on that road. Thus, the directed graph 700 includes information at different levels of granularity. In an embodiment, the directed graph with high granularity is also a subgraph of another directed graph with a larger scale. For example, a directed graph where the starting point 702 and the ending point 704 are far apart (e.g., many miles apart) has most of its information at a low granularity, and this directed graph is based on stored data, but it also includes some high-granularity information for representing a portion of the physical location in the field of view of the vehicle 100.

[0101] Nodes 706a-706d are different from objects 708a-708b, which cannot overlap with nodes. In embodiments, at low granularity, objects 708a-708b represent areas that a vehicle cannot traverse, e.g., areas without streets or roads. At high granularity, objects 708a-708b represent physical objects in the field of view of vehicle 100, e.g., other vehicles, pedestrians, or other entities with which vehicle 100 cannot share physical space. In embodiments, some or all of objects 708a-708b are static objects (e.g., objects that do not change position, such as streetlights or utility poles) or dynamic objects (e.g., objects that can change position, such as pedestrians or other cars).

[0102] Nodes 706a-706d are connected by edges 710a-710c. If two nodes 706a-706b are connected by edge 710a, vehicle 100 can travel between one node 706a and the other node 706b, e.g., without having to travel to an intermediate node before reaching the other node 706b. (When referring to vehicle 100 traveling between nodes, it means that vehicle 100 travels between two physical locations represented by the respective nodes.) Edges 710a-710c are typically bidirectional, in the sense that vehicle 100 can travel from a first node to a second node, or from the second node to the first node. In embodiments, edges 710a-710c are unidirectional, in the sense that vehicle 100 can travel from a first node to a second node, but vehicle 100 cannot travel from the second node to the first node. Edges 710a-710c are unidirectional in cases where they represent, e.g., one-way streets, individual lanes of a street, road, or highway, or other features that can only be traversed in one direction due to legal or physical constraints.

[0103] In embodiments, planning system 404 uses directed graph 700 to identify a path 712 consisting of nodes and edges between start node 702 and end node 704.

[0104] Edges 710a-710c have associated costs 714a-714b. Costs 714a-714b represent the resources that would be spent if vehicle 100 selected that edge. A typical resource is time. For example, if the physical distance represented by one edge 710a is twice the physical distance represented by another edge 710b, then the associated cost 714a of the first edge 710a can be twice the associated cost 714b of the second edge 710b. Other factors affecting time include anticipated traffic, the number of intersections, speed limits, etc. Another typical resource is fuel economy. Two edges 710a-710b can represent the same physical distance, but due to factors such as road conditions and anticipated weather, one edge 710a may require more fuel than the other edge 710b.

[0105] When planning system 404 identifies path 712 between start point 702 and end point 704, planning system 404 typically selects a path that is optimized for cost, such as the path that has the minimum total cost when the individual costs of the edges are added together.

[0106] Autonomous Vehicle Control

[0107] Figure 8 Show (for example, as) Figure 4 The diagram shows a block diagram 800 of the inputs and outputs of the control system 406. The control system operates according to a controller 802, which includes, for example, one or more processors similar to processor 304 (e.g., one or more computer processors such as microprocessors or microcontrollers or both); short-term and / or long-term data storage devices similar to main memory 306, ROM 308, and storage device 310 (e.g., memory area random access memory or flash memory or both); and instructions stored in the memory that, when executed (e.g., by one or more processors), perform the operation of controller 802.

[0108] In one embodiment, controller 802 receives data representing a desired output 804. The desired output 804 typically includes speed, such as rate and heading. The desired output 804 may be based, for example, from (e.g., as...) Figure 4The data received by the planning system 404 (shown) is used by the controller 802 to generate data that can be used as throttle input 806 and steering input 808. The throttle input 806 represents, for example, engaging the throttle (e.g., acceleration control) of the vehicle 100 by engaging a steering pedal or engaging another throttle control to achieve the magnitude of the desired output 804. In some examples, the throttle input 806 also includes data that can be used to engage the brakes (e.g., deceleration control) of the vehicle 100. The steering input 808 represents the steering angle, for example, the steering control (e.g., steering wheel, steering angle actuator, or other functionality used to control the steering angle) of the vehicle should be positioned to achieve the angle of the desired output 804.

[0109] In embodiments, the controller 802 receives feedback that is used in adjusting the input provided to the throttle and steering. For example, if the vehicle 100 encounters an interference 810, such as a hill, the measured velocity 812 of the vehicle 100 drops below the desired output velocity. In embodiments, any measured output 814 is provided to the controller 802 so that the required adjustments are made, for example, based on the difference 813 between the measured velocity and the desired output. The measured output 814 includes measured position 816, measured velocity 818 (including both speed and heading), measured acceleration 820, and other outputs measurable by sensors of the vehicle 100.

[0110] In embodiments, information about the interference 810 is detected in advance, for example, by sensors such as cameras or LiDAR sensors, and the information is provided to a predictive feedback system 822. The predictive feedback system 822 then provides information to the controller 802 that the controller 802 can use to adjust accordingly. For example, if a sensor of the vehicle 100 detects (“sees”) a hill, the controller 802 can use that information to prepare to engage the throttle at the appropriate time to avoid a significant deceleration.

[0111] Figure 9 A block diagram 900 showing the inputs, outputs, and components of the controller 802. The controller 802 has a velocity analyzer 902 that influences the operation of a throttle / brake controller 904. For example, the velocity analyzer 902 instructs the throttle / brake controller 904 to use a throttle / brake 906 to accelerate or to decelerate, based on feedback, for example, received by the controller 802 and processed by the velocity analyzer 902.

[0112] The controller 802 also has a lateral tracking controller 908 that influences the operation of a steering wheel controller 910. For example, the lateral tracking controller 908 instructs the steering wheel controller 910 to adjust the position of a steering angle actuator 912, based on feedback, for example, received by the controller 802 and processed by the lateral tracking controller 908.

[0113] Controller 802 receives several inputs for determining how to control the throttle / brake 906 and steering angle actuator 912. Planning system 404 provides controller 802 with information, for example, to select the heading of vehicle 100 at the start of operation and to determine which road segment vehicle 100 will cross when it reaches an intersection. Positioning system 408 provides controller 802 with information describing the current location of vehicle 100, for example, so that controller 802 can determine whether vehicle 100 is at the expected location based on positive control of the throttle / brake 906 and steering angle actuator 912. In embodiments, controller 802 receives information from other inputs 914, such as information received from a database, computer network, etc.

[0114] Autonomous Vehicle Notification and Post-Action Explanation System

[0115] Figure 10 A block diagram of an example autonomous vehicle notification and post-action interpretation system 1000 is shown. In this embodiment, the autonomous vehicle notification and post-action interpretation system 1000 is... Figure 3 This is part of the processor 304 shown. In this embodiment, the autonomous vehicle notification and post-action interpretation system 1000 is... Figure 2 This is part of the processor in the cloud 202 shown. Typically, the autonomous vehicle notification and post-action interpretation system 1000 identifies at least one reason why the vehicle deviates from its planned path. Generally, deviation can be any unexpected movement or behavior of the vehicle. In this example, the reason for the vehicle deviating from the planned path is an object (e.g., a natural obstacle 191, a vehicle 193, a pedestrian 192, a cyclist, or other obstacles). The reason for the deviation is reported to the user in the form of a visual or auditory notification.

[0116] The autonomous vehicle post-action interpretation system 1000 takes deviation signal 1011, planning data 1012, tracked object 1013, and environmental data 1014 as inputs. Typically, deviation signal 1011 indicates that the vehicle has deviated from its planned path. Deviation signal 1011 can be generated after the vehicle has deviated from its planned path. Example deviation signal 1011 is a collision avoidance signal generated by the collision avoidance subsystem of AV stack 1016. Another example deviation signal 1011 is a collision avoidance signal generated by safety system 1018. An additional example deviation signal 1011 is a drift signal generated by the drift subsystem of AV stack 1016.

[0117] The collision avoidance subsystem is a subsystem of the AV stack 1016 that is specialized to perform vehicle maneuvering actions to avoid collisions with objects or obstacles in the vicinity of the vehicle. In embodiments, the collision avoidance subsystem is part of the control circuit 406. An example collision avoidance signal generated by the collision avoidance subsystem or the safety system 1018 is to decelerate to avoid a collision with a suddenly appearing pedestrian.

[0118] The drift subsystem is a subsystem of the AV stack 1016 that is used to cause the vehicle to steer to at least maintain a predefined distance (e.g., 1 meter) from a nearby object (e.g., a truck) even if there is no current or future risk of a collision with the nearby object. In embodiments, the predefined distance is a lateral distance from the vehicle (e.g., from a side of the vehicle) to the nearby object. In embodiments, the drift subsystem is part of the planning system 404. In an example, if a large truck occupies the right side of a lane, the drift subsystem causes the vehicle to steer from the center of the lane to the left side of the lane. In this example, a drift signal generated by the drift subsystem is to cause the vehicle to steer to the left to maintain a lateral distance from the large truck. In embodiments, the drift signal causes the vehicle to steer to make a lane change to maintain the predefined distance from the nearby object. In embodiments, after the vehicle makes a lane change to maintain the predefined distance from the nearby object, the drift signal causes the vehicle to steer to return to the lane in which the vehicle was previously traveling.

[0119] In general, the planning data 1012 represents data associated with instructions for causing the vehicle to navigate from a first point toward a second point. An example set of planning data 1012 is data used and / or generated by the planning circuit 404 and / or the control circuit 406 of the AV 100. In an example, the tracked objects 1013 are objects in the environment that have been detected and are being monitored by the autonomous driving system 101, the AV stack 1016, the safety system 1018, or any combination thereof. For example, the tracked objects are determined by the perception system (e.g., the perception system 402 of the AV 100) and are labeled according to their respective classifications. Figure 4

[0120] The environment data 1014 represents data associated with the surrounding environment of the vehicle. An example set of environment data 1014 includes a semantically drivable surface and / or labeled objects generated based on sensor data obtained from on-board sensors. In embodiments, the environment data 1014 is generated by identifying and labeling objects and / or surfaces in sensor data (such as the outputs 504a-504d shown in FIG. 5, etc.). In embodiments, the identifying and labeling uses at least one neural network, such as a VoxNet, a PointNet, a SegNet, and / or a YOLO, etc. Figure 5

[0121] According to​​Figure 11 Further details regarding the simulation based on the deviation signal 1011, the planning data 1012, the tracked objects 1013, the environment data 1014, and / or components thereof are described below.

[0122] In embodiments, the deviation signal 1011, the planning data 1012, the tracked objects 1013, and the environment data 1014 are generated by the autonomous driving system 1010. In embodiments, the autonomous driving system 1010 includes an AV stack 1016 and a safety system 1018. In embodiments, the autonomous driving system 1010 is part of the AV system 120 shown in FIG. 1. Figure 1 In embodiments, the autonomous driving system 1010 includes the perception system 402 and / or the localization system 408 shown in FIG. 1. In embodiments, the AV stack 1016 includes the planning circuit 404 and / or the control circuit 406 shown in FIG. 1. Figure 4 In embodiments, the autonomous driving system 1010 includes the perception system 402 and / or the localization system 408 shown in FIG. 1. In embodiments, the AV stack 1016 includes the planning circuit 404 and / or the control circuit 406 shown in FIG. 1. Figure 4 In embodiments, the autonomous driving system 1010 includes the perception system 402 and / or the localization system 408 shown in FIG. 1. In embodiments, the AV stack 1016 includes the planning circuit 404 and / or the control circuit 406 shown in FIG. 1.

[0123] In embodiments, the safety system 1018 is one or more auxiliary systems that are at least partially independent of the AV stack 1016 to ensure safety of the autonomous driving system 1010. In such embodiments, the safety system 1018 is a subsystem that monitors objects in the vicinity of the AV and reacts to these objects, operating independently of the planning circuit 404. In embodiments, the safety system 1018 applies base level control commands (e.g., braking and / or turning, etc.) directly to the vehicle. The safety system 1018 can be a standalone system that is separate from and operates independently of the AV stack. An example safety system is an automatic emergency braking (AEB) system. The AEB system uses sensors to detect a forward obstacle and assess whether a collision is likely to occur. If a collision is likely to occur, the AEB system will apply the brakes. In embodiments, the safety system 1018 includes standalone sensors. In embodiments, the safety system 1018 uses sensors of the autonomous vehicle, such as the LiDAR 502a, RADAR 502b, or camera 502c, etc. shown in FIG. 1. Figure 5 In embodiments, the safety system 1018 is one or more auxiliary systems that are at least partially independent of the AV stack 1016 to ensure safety of the autonomous driving system 1010. In such embodiments, the safety system 1018 is a subsystem that monitors objects in the vicinity of the AV and reacts to these objects, operating independently of the planning circuit 404. In embodiments, the safety system 1018 applies base level control commands (e.g., braking and / or turning, etc.) directly to the vehicle. The safety system 1018 can be a standalone system that is separate from and operates independently of the AV stack. An example safety system is an automatic emergency braking (AEB) system. The AEB system uses sensors to detect a forward obstacle and assess whether a collision is likely to occur. If a collision is likely to occur, the AEB system will apply the brakes. In embodiments, the safety system 1018 includes standalone sensors. In embodiments, the safety system 1018 uses sensors of the autonomous vehicle, such as the LiDAR 502a, RADAR 502b, or camera 502c, etc. shown in FIG. 1.

[0124] In embodiments, the planning data 1012 is generated using the AV stack 1016, while the deviation signal 1011 is generated from the AV stack 1016 or the safety system 1018. In embodiments, the planning data 1012 is generated at least in part using the planning circuit 404 as shown in FIG. 1. Figure 4 In embodiments, the planning data 1012 is generated using the AV stack 1016, while the deviation signal 1011 is generated from the AV stack 1016 or the safety system 1018. In embodiments, the planning data 1012 is generated at least in part using the planning circuit 404 as shown in FIG. 1.

[0125] In embodiments, the environment data 1014 represents the environment in which the vehicle operates. In embodiments, the environment data 1014 is generated by the autonomous driving system 1010. In such embodiments, the environment data 1014 is generated at least in part by the perception system 402. In embodiments, the environment data 1014 represents the environment in which the vehicle operates. In embodiments, the environment data 1014 is generated by the autonomous driving system 1010. In such embodiments, the environment data 1014 is generated at least in part by the perception system 402.

[0126] The components of the deviation signal 1011, the planning data 1012, the tracked objects 1013, and the environment data 1014 (in some cases after processing) are provided as inputs to a simulation system 1020. The simulation system 1020 determines the responsible object(s) 1022 that caused the deviation. More details regarding the components of the deviation signal 1011, the planning data 1012, and the environment data 1014, the simulation system 1020, and the determination of the responsible objects 1022 are explained below with reference to Figure 11

[0127] Once the responsible objects 1022 that caused the deviation are determined, the responsible objects 1022 are provided as inputs to a notification system 1030. The notification system 1030 takes the responsible objects 1022 and searches for the responsible objects 1022 in the recent history of sensor data. For example, if the responsible objects 1022 are squirrels that crossed the travel lane in which the vehicle is traveling, the notification system 1030 searches for the squirrels in the recent history of camera images (e.g., images captured in the past 10 seconds) and generates a set of images or a video showing the squirrels. In embodiments, the sensor data is unprocessed (e.g., unlabelled) and the notification system 1030 uses a neural network such as YOLO or PointNet to detect and identify the responsible objects 1022 in the sensor data. In embodiments, processed (e.g., labelled) sensor data can be obtained from an object detection neural network that is configured to receive sensor data and process the sensor data to detect at least one object in the 3D space surrounding the sensor (e.g., a natural obstacle 191, a vehicle 193, and a pedestrian 192; a cyclist; and other obstacles). In embodiments, the object detection neural network is a feedforward convolutional neural network that takes the outputs 504a-504d (e.g., sensor data), generates a set of bounding boxes for potential objects in the 3D space, and generates a confidence score for the existence of an object class instance (e.g., a car, a pedestrian, or a bicycle) within the bounding box. In embodiments, the object detection network is a semantic segmentation neural network such as SegNet. In examples, SegNet takes a set of images as input, predicts the class of each pixel in the images, and outputs semantic segmentation data (e.g., labels) for each pixel in the images. Figure 1

[0128] ​​The notification system 1030 searches the processed sensor data for a matching label. For example, if a semantic segmentation mask is available for the image captured by the camera (e.g., stored in a database) and the responsible object 1022 is labeled as a squirrel, the notification system 1030 searches the stored segmentation mask for a region labeled as a squirrel and then locates the corresponding image captured by the camera showing the squirrel. In embodiments, the responsible object 1022 causing the deviation is static, such as a fallen tree or a parked car. In embodiments, the responsible object 1022 causing the deviation is dynamic, such as a squirrel or a pedestrian.

[0129] The notification system 1030 generates a compiled message 1032. In embodiments, the compiled message 1032 includes a warning. In embodiments, the compiled message 1032 identifies the responsible object 1022 causing the deviation. In embodiments, the compiled message 1032 is an audio message. In embodiments, the compiled message 1032 is a video message. In embodiments, the compiled message 1032 includes a distance measurement between the vehicle and the responsible object 1022. For example, the compiled message 1032 can indicate the distance in meters or in feet from the vehicle to the responsible object 1022. In embodiments, if the compiled message 1032 is a video message, the compiled message 1032 can include an animation showing the deviation of the vehicle from the planned path. In embodiments, if the compiled message 1032 is a video message, the compiled message 1032 includes video captured by at least one camera of the vehicle with annotated augmented reality (AR) overlays.

[0130] The broadcast system 1040 takes the compiled message 1032 as input and presents the compiled message 1032 to one or more passengers in the vehicle (such as a passenger) via one or more interfaces based on the passenger’s preferences. In embodiments, if the passenger prefers an audio notification, the broadcast system 1040 delivers the compiled message 1032 via an available audio interface such as an in-vehicle speaker or a passenger’s head-mounted headphones. In embodiments, if the passenger prefers a visual notification, the broadcast system 1040 delivers the compiled message 1032 via an available video interface such as an in-vehicle screen or a passenger’s personal smartphone. In embodiments, the compiled message 1032 is a video message augmented by an audio warning or explanation. In embodiments, the passenger has the option to turn off the broadcast system 1040’s messaging, which prevents the broadcast system from delivering messages to the passenger. As a result, the passenger is prevented from receiving any notification communicated by the messages output by the broadcast system. In embodiments, the broadcast system 1040 transmits the compiled message 1032 to a remote operator (e.g., a web-based operator) upon detecting the deviation.

[0131] In embodiments, after a deviation occurs, the compiled message 1032 is saved to a log or database. In embodiments, the compiled message 1032 includes a timestamp, metadata (e.g., a header or data size indication), and / or an AV state, among others. In embodiments, the log or database is Figure 4 part of the database system 410 shown.

[0132] Figure 11 A block diagram 1100 of an example simulation system 1020 used in the autonomous vehicle action post-explanation system 1000 shown is shown. Figure 10 The simulation system 1020 determines at least one object responsible for the deviation based on the deviation signal 1011, planning data 1012, and / or environment data 1014.

[0133] In embodiments, the deviation signal 1011 is a collision avoidance signal indicating that the vehicle is making a maneuvering action to avoid a collision with a responsible object 1022. In embodiments, the deviation signal 1011 is a drift signal indicating that the vehicle is drifting from the planned path 1112. In embodiments, the deviation signal 1011 contains signature information indicating where it originated or was generated (such as from the AV stack 1016 or the safety system 1018, among others). Figure 10 In such embodiments, the signature information of the deviation signal 1011 further indicates the subsystem from which the deviation signal 1011 originated, such as a collision avoidance subsystem or a drift subsystem of the AV stack 1016, among others.

[0134] In embodiments, the planning data 1012 includes an AV state 1110 (such as an AV pose, or a position and orientation of the AV, among others), and a planned path 1112 (which is a path planned prior to the deviation).

[0135] In embodiments, the environment data 1014 includes a semantic drivable surface 1114, which is a semantic map indicating drivable surfaces (such as marked travel lanes and / or intersections, among others) in the environment. In embodiments, the environment data 1014 includes tracked objects 1118. In embodiments, the tracked objects 1118 are associated with object classes (such as cars, pedestrians, squirrels, and / or construction cones, among others) and relative positions with respect to the vehicle. In embodiments, the tracked objects 1118 include background objects, such as distant buildings and / or vegetation, among others. In embodiments, the semantic drivable surface 1114 and the tracked objects 1118 are generated by identifying and labeling objects in sensor data (such as the outputs 504a-504d of the perception system 402, among others). Figure 5 In embodiments, the identification and labeling of objects is performed by the perception system 402 using neural networks (such as VoxNet, PointNet, SegNet, and / or YOLO, among others).

[0136] The tracked objects 1118 are provided to a tracked selector and iterator 1120, which is configured to iterate through the tracked objects 1118 and generate a selected object 1122. In embodiments, the tracked objects 1118 form a queue in the tracked selector and iterator 1120, where the queue is determined by the distance between the objects and the vehicle (e.g., 2D Euclidean distance and / or 2D Manhattan distance, etc.). In embodiments, the queue is implemented using a standard data structure, such as a queue, stack, heap, list, array, and / or binary search tree, etc. In embodiments, objects that are closer to the vehicle have a higher priority in the queue. In embodiments, additionally, the tracked selector and iterator 1120 can use properties of the tracked objects 1118 (e.g., velocity, speed, acceleration, activity) to reduce the size of the queue by removing objects that are less likely to be the cause of the deviation, such as inactive (e.g., stationary) and / or distant objects, etc. In embodiments, the selected object 1122 is the highest priority object that is taken out and removed from the queue. In embodiments, in the event that the queue is exhausted and no responsible object 1022 is found, a new queue is constructed by increasing the search range from the vehicle. For example, in the event that the queue of tracked objects 1118 within 50 meters of the vehicle is exhausted, a new queue of tracked objects 1118 that are 50 to 100 meters away from the vehicle is constructed. In such embodiments, objects in the new queue can be removed from the new queue by the tracked selector and iterator 1120 based on properties of the object (e.g., in the event that the object has 0 velocity and 0 acceleration). In embodiments, the queue is formed from all tracked objects 1118 before objects are removed from the queue based on properties.

[0137] The system selector 1130 determines a selection signal 1132 from the deviation signal 1011. In embodiments, the system selector 1130 extracts signature information from the deviation signal 1011. In such embodiments, the signature information of the selection signal 1132 indicates the origin of the deviation signal 1011 (e.g., where the deviation signal 1011 was generated). In embodiments, the deviation signal 1011 from the collision avoidance subsystem or drift subsystem of the AV stack 1016 has a different number and / or type of components than the deviation signal 1011 from the safety system 1018. In such embodiments, the number and / or type of components of the deviation signal 1011 is the signature information. In embodiments, the system selector 1130 determines the origin of the deviation signal 1011 based on the signature information, such as the collision avoidance subsystem or drift subsystem of the AV stack 1016, or the safety system 1018, etc. In embodiments, the selection signal 1132 is used to activate a corresponding subsystem of the simulation system 1020.

[0138] The simulation system 1020 includes subsystems such as a first set of simulators 1150 and a second set of simulators 1160. In embodiments, the first set of simulators 1150 includes a collision avoidance simulator 1152 that utilizes the functionality of the collision avoidance subsystem of the AV stack 1016 to simulate motion (e.g., behavior) of the vehicle. In embodiments, the first set of simulators 1150 includes a drift simulator 1154 that utilizes the functionality of the drift subsystem of the AV stack 1016 to simulate motion of the vehicle. In embodiments, the second set of simulators 1160 includes a safety system simulator 1162 that utilizes at least one function of the safety system 1018 to simulate motion of the vehicle. Based on the results of the simulation, the simulation system 1020 determines a responsible object 1022.

[0139] The simulation system 1020 takes as inputs the AV state 1110, the planned path 1112, the semantically drivable surface 1114, the selection signal 1132, and the selected object 1122. The selection signal 1132 determines which subsystem of the simulation system 1020 to activate based on the type and / or origin of the indicated deviation signal 1011. In an example, if the deviation signal 1011 is a collision avoidance signal generated by the collision avoidance subsystem of the AV stack 1016, the selection signal 1132 activates the collision avoidance simulator 1152 in the first set of simulators 1150. In another example, if the deviation signal 1011 is a drift signal generated by the drift subsystem of the AV stack 1016, the selection signal 1132 activates the drift simulator 1154 in the first set of simulators 1150. In yet another example, if the deviation signal 1011 is a collision avoidance signal generated by the safety system 1018, the selection signal 1132 activates the safety system simulator 1162 in the second set of simulators 1160. In embodiments, each simulator has a backup simulator to ensure the main simulation system 1020 operates in the event of a subsystem failure. For example, if the selection signal 1132 activates the drift simulator 1154 but the drift simulator 1154 is unavailable, a backup drift simulator is activated instead, which performs the same functionality as the drift simulator 1154.

[0140] For each selected object 1122, a subsystem such as the first set of simulators 1150 or the second set of simulators 1160 of the simulation system 1020 activates based on the selection signal 1132 and simulates a trajectory of the vehicle based on the AV state 1110, the planned path 1112, and the semantically drivable surface 1114. A simulation environment is constructed based on the semantically drivable surface 1114. Each simulation generates a trajectory of the vehicle in the simulation environment based on the AV state 1110 and the currently selected object 1122. The simulated trajectory in each simulation represents the motion of the vehicle as if only the currently selected object 1122 existed in the simulation environment. The simulated trajectory generated in each simulation is compared to the planned path 1112. If the simulated trajectory differs from the planned path 1112, the divergence in the simulated trajectory is reported. The selected object 1122 that caused the divergence is determined to be the responsible object 1022.

[0141] If the currently selected object 1122 fails to produce a divergence in the simulated trajectory, the next object in the queue formed by the tracking selector and iterator 1120 is selected as the new selected object 1122. If the current queue is exhausted, the first object in a new queue formed by the tracking selector and iterator 1120 is selected as the selected object 1122. The system iterates until the selected object 1122 is determined to be the responsible object 1022 that caused a divergence in the simulated trajectory of the vehicle. In embodiments, the simulations are performed at least in part with a remote processor external to the vehicle, such as a processor in the cloud 202. Figure 2 The responsible object 1022 is then provided as input to the notification system 1030.

[0142] In embodiments, the responsible object 1022 determined when simulating with the collision avoidance simulator 1152 is an object that is close to the vehicle or its planned path, such as a pedestrian crossing the street. In embodiments, the responsible object 1022 determined when simulating with the drift simulator 1154 is an object such as a construction cone placed on the road.

[0143] Figure 12 A flowchart illustrating a process 1200 for notifying and explaining the actions of a vehicle is shown. In embodiments, the vehicle is the AV 100 shown. Figure 1 In implementations, the process 1200 is performed by a processor such as the processor 304 shown. Figure 3 In implementations, the process 1200 is performed by the perception system 402, the planning system 404, the control system 406, or the localization system 408 shown. Figure 4 In implementations, the process 1200 is performed by a processor such as the processor 304 shown.

[0144] At block 1202, a planned path of the vehicle, a state of the vehicle, and environmental data of an environment in which the vehicle is operating are received. In implementations, the planned path of the vehicle is Figure 11 the planned path 1112 shown. In implementations, the state of the vehicle is Figure 11 the AV state 1110 shown. In implementations, the environmental data of the environment in which the vehicle is operating is Figure 10 and Figure 11 the environmental data 1014 shown. In embodiments, the environmental data 1014 includes a semantic drivable surface 1114, which is a semantic map indicating drivable surfaces in the environment. In embodiments, the environmental data 1014 includes a tracked object 1118, which is a recognized object in the environment.

[0145] At block 1204, a deviation signal is received. In implementations, the deviation signal is Figure 10 and Figure 11 the deviation signal 1011 shown. In implementations, the deviation signal 1011 is a drift signal indicating that the vehicle is drifting from the planned path. In implementations, the deviation signal 1011 is a collision avoidance signal indicating that the vehicle is performing a maneuver to avoid a collision with at least one static or dynamic object in the environment.

[0146] At block 1206, a determination is made as to whether the deviation signal is reported by a first system or a second system of the vehicle. In implementations, the first system of the vehicle is Figure 10 the AV stack 1016 shown. In implementations, the second system of the vehicle is Figure 10 the safety system 1018 shown. In implementations, the determination is made by Figure 11 the system selector 1130 shown. In response, at block 1208, a first set of simulators or a second set of simulators for simulating the vehicle in the environment is activated. In implementations, the activation is based on Figure 11 the selection signal 1132 shown. In implementations, the first set of simulators is Figure 11 the first set of simulators 1150 shown and the second set of simulators is Figure 11 the second set of simulators 1160 shown. In implementations, as Figure 11 shown, the first set of simulators includes a collision avoidance simulator 1152 for simulating collision avoidance and a drift simulator 1154 for simulating the vehicle drifting from the planned path. In implementations, as Figure 11 shown, the second set of simulators includes at least a safety system simulator 1162 for simulating at least one function of the safety system.

[0147] At block 1210, the vehicle in the environment is simulated using the selected first set of simulators or the second set of simulators. The simulation includes simulating the vehicle and at least one static object or dynamic object. In implementations, the simulation is performed by the simulation system 1020 shown in Figure 10 and Figure 11 In implementations, the simulation is performed at least in part by a remote processor external to the vehicle, such as the cloud 202 shown in Figure 2 In implementations, the at least one static object or dynamic object is obtained from the object selector 1120 shown in Figure 11

[0148] At block 1212, a message is generated based on the results of the simulation. In implementations, the message is the compiled message 1032 shown in Figure 10 In implementations, the compiled message 1032 contains audio, video with or without AR overlays identifying the responsible object 1022 causing the deviation. In implementations, the responsible object 1022 is a static object, such as a construction cone. In implementations, the responsible object 1022 is a dynamic object, such as a pedestrian.

[0149] At block 1214, the message is presented to at least one occupant of the vehicle. In implementations, the at least one occupant of the vehicle is a passenger. In implementations, the message is an audio message broadcast through an audio interface. In implementations, the message is a video message and the presentation is through a display system of the vehicle.

[0150] In the foregoing description, embodiments of the application have been described with reference to a number of specific details that can vary depending on implementation. Thus, the description and drawings are to be regarded as illustrative rather than restrictive. The sole and exclusive indicator of the scope of the application, and what is intended by the applicants to be the scope of the application, is set forth in the claims issued from this application, including any subsequent correction. Any definitions of terms set forth in this detailed description are expressly incorporated herein by reference to the meaning attributed in the claims. In addition, any priority documents and / or articulated definitions of terms in the priority documents are expressly incorporated herein by reference in their entirety. In addition, as used in this specification and claims, the singular forms "a," "an" and "the" include plural referents unless the content clearly dictates otherwise.​

Claims

1. A method for a vehicle, comprising: Using at least one processor, the planned path of the vehicle, the status of the vehicle, and environmental data of the environment in which the vehicle is operating are received; Using the at least one processor, the deviation signal is received; Using the at least one processor, it is determined whether the deviation signal was reported by a first system or a second system of the vehicle; Since the deviation signal was reported by the first system of the vehicle, Using the at least one processor, a first set of simulators is selected for simulating a first behavior of the vehicle in the environment; The deviation signal was reported by the second system of the vehicle. Using the at least one processor, a second set of simulators is selected for simulating a second behavior of the vehicle in the environment; Using the at least one processor, the first or second behavior of the vehicle is simulated using a selected first set of simulators or a second set of simulators to determine if a currently selected object is deviating in the simulation, wherein the currently selected object is iteratively selected by an iterator until the currently selected object is the responsible object causing the deviation; Using the at least one processor, a message is generated that identifies the responsible party that caused the deviation; as well as The message is presented to at least one occupant of the vehicle using the at least one processor.

2. The method according to claim 1, wherein, The deviation signal is a drift signal indicating that the vehicle is drifting away from the planned path.

3. The method according to claim 1 or 2, wherein, The deviation signal is a collision avoidance signal, which indicates that the vehicle is performing a maneuver to avoid colliding with at least one static or dynamic object in the environment.

4. The method according to claim 1 or 2, wherein, The environmental data includes a semantic map indicating drivable surfaces in the environment.

5. The method according to claim 1 or 2, wherein, The message is an audio message broadcast to one or more occupants of the vehicle.

6. The method according to claim 1 or 2, wherein, The message is a video message presented to the at least one occupant of the vehicle using the vehicle's display system.

7. The method according to claim 1 or 2, wherein, The message contains information identifying the static or dynamic object that caused the deviation.

8. The method according to claim 7, wherein, The message includes distance measurement results between the vehicle and the static or dynamic object.

9. The method according to claim 1 or 2, wherein, The message was transmitted to the remote operator.

10. The method according to claim 1 or 2, wherein, The message is stored in a log or database.

11. The method according to claim 1 or 2, wherein, The first set of simulators includes a first simulator for simulating collision avoidance and a second simulator for simulating the vehicle drifting from the planned path.

12. The method according to claim 1 or 2, wherein, The second set of simulators includes at least a third simulator for simulating at least one function of a security system.

13. The method according to claim 1 or 2, wherein, The simulation was performed, at least in part, using a remote processor located outside the vehicle.

14. The method according to claim 6, wherein, The video message includes an animation showing the vehicle deviating from the planned path.

15. The method according to claim 6, wherein, The video message includes video with labeled augmented reality overlay, i.e. labeled AR overlay, captured by at least one camera of the vehicle.

16. The method according to claim 1 or 2, wherein, The process includes, before presenting the message to the at least one occupant: It was determined that at least one occupant did not select the option to disable messaging.

17. The method according to claim 1 or 2, wherein, The first system is a planning system, and the second system is a security system.

18. A method for a vehicle, comprising: Using at least one processor, data is received indicating the planned path of the vehicle, the current state of the vehicle, and the environment in which the vehicle operates; Using the at least one processor, a first deviation signal or a second deviation signal is received; Using the at least one processor, the tracked object is iterated to generate the currently selected object for each simulation; Based on the received first deviation signal Using a first simulator with the data as input, the movement of the vehicle and the currently selected object are simulated to determine whether the currently selected object is the responsible party that caused the vehicle to deviate from the planned path. Based on receiving the second deviation signal Using a second simulator with the data as input, the movement of the vehicle and the currently selected object are simulated to determine whether the currently selected object is the responsible party that caused the vehicle to deviate from the planned path. as well as Using the at least one processor, a message is presented to the occupants of the vehicle, the message indicating a deviation from the planned path and the responsible party identified as the cause of the deviation.

19. The method of claim 18, further comprising: Using the at least one processor, determine whether the first emulator is unavailable; Based on the fact that the first simulator is unavailable and the first deviation signal has been received. Using a backup first simulator with the data as input, the movement of the vehicle and objects are simulated to determine whether the objects caused the vehicle to deviate from the planned path; as well as Based on the fact that the second simulator is unavailable and the second deviation signal has been received. Using a backup second simulator that takes the data as input, the movement of the vehicle and objects are simulated to determine whether the objects caused the vehicle to deviate from the planned path.

20. The method according to claim 18, wherein, The first simulator simulates the vehicle drifting from the planned path, and the second simulator simulates the vehicle avoiding collisions with the object.

21. A vehicle system, comprising: At least one processor; as well as A memory for storing instructions that, when executed by the at least one processor, cause the at least one processor to perform the method according to any one of claims 1 to 20.

22. A computer program product comprising computer instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 20.

Images