Systems, methods, and computer-readable storage media for a vehicle

Abstract

The invention relates to a system, method, and computer-readable storage medium for a vehicle. Therein, the following techniques are described: receiving, from at least one sensor of a vehicle, a sensor measurement indicative of at least one of a sound and a vibration associated with a road element; identifying the road element based on a pattern in the sensor measurement; determining a vehicle behavior of the vehicle based on the road element; and controlling the vehicle to operate in accordance with the vehicle behavior.

Description

TECHNICAL FIELD

[0001] This specification relates to surface guided vehicle behavior. BACKGROUND

[0002] Roads on which vehicles are driven include various surface elements. For example, roads include pavement markings, road dividers, painted road markings, parking blocks / chalk marks / blocks, and rumble strips. SUMMARY

[0003] According to an aspect of the present application, a system for a vehicle, comprising: at least one sensor of the vehicle; at least one computer readable medium storing computer executable instructions; at least one processor configured to execute the computer executable instructions, the execution operative to perform operations comprising: receiving, from the at least one sensor, sensor measurements indicative of at least one of sound and vibration associated with a road element; identifying the road element based on a pattern in the sensor measurements; determining a vehicle behavior for the vehicle based on the road element; and controlling the vehicle to operate in accordance with the vehicle behavior.

[0004] According to another aspect of the present application, a method for a vehicle, comprising: receiving, from at least one sensor of the vehicle, sensor measurements indicative of at least one of sound and vibration associated with a road element; identifying the road element based on a pattern in the sensor measurements; determining a vehicle behavior for the vehicle based on the road element; and controlling the vehicle to operate in accordance with the vehicle behavior.

[0005] According to yet another aspect of the present application, a non-transitory computer readable storage medium comprising at least one program, the at least one program being executable by at least one processor of a first device, the at least one program comprising instructions that, when executed by the at least one processor, cause the first device to perform the above method. BRIEF DESCRIPTION OF DRAWINGS

[0006] Figure 1 An example of an autonomous vehicle with autonomous capabilities is shown.

[0007] Figure 2 An example "cloud" computing environment is shown.

[0008] Figure 3 A computer system is shown.

[0009] Figure 4 An example architecture of an autonomous vehicle is shown.

[0010] Figure 5A block diagram showing the relationship between inputs and outputs of the planning module.

[0011] Figure 6 A block diagram showing the inputs and outputs of the control module.

[0012] Figure 7 A block diagram illustrating an example system for surface-guided decision making.

[0013] Figure 8 A block diagram illustrating an example system for localization correction and sensor damage detection.

[0014] Figure 9 A flowchart showing a process for surface-guided decision making. DETAILED DESCRIPTION

[0015] In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present application. It will be apparent, however, to one skilled in the art that the present application can be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form in order to avoid unnecessarily obscuring the present application.

[0016] In the drawings, for purposes of clarity, specific arrangements or orderings of elements are shown. However, it is to be understood that such elements can be arranged or ordered differently or removed altogether, without departing from the scope of the application. Further, some elements shown are optional and can not be included, or can be included with other elements, in some embodiments.

[0017] Further, in the drawings, connecting elements, such as lines or arrows or the like, can represent some of the worst possible combinations of elements for illustrating relationships or associations between other elements. No connection or relationship or association between elements should be read as necessarily implying that there is some other, additional connection or relationship or association between elements.

[0018] Reference will now be made in detail to embodiments, examples of which are illustrated in the accompanying drawings. In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the various embodiments described. However, it will be apparent to one skilled in the art that the various embodiments described can be practiced without these specific details. In other instances, well-known methods, procedures, components, circuits, and networks have not been described in detail so as not to unnecessarily obscure aspects of the embodiments.

[0019] Several features described below can each be used independently of each other or with any combination of other features. However, any individual feature can not address any of the problems discussed above or can only address one of the problems discussed above. Some of the problems discussed above can not be fully addressed by any one feature described herein. Although a heading is provided, information related to a particular heading can also be found elsewhere in the specification, not necessarily in the section with the heading. Embodiments are described herein according to the following outline:

[0020] 1. Overall Summary

[0021] 2. System Overview

[0022] 3. Autonomous Vehicle Architecture

[0023] 4. Autonomous Vehicle Inputs

[0024] 5. Autonomous Vehicle Planning

[0025] 6. Autonomous Vehicle Control

[0026] 7. Surface Guidance Decision Making

[0027] Overall Summary

[0028] Adapting behavior of a vehicle based on identifying surface elements that affect safety and driving functionality of the vehicle. To identify surface elements, vehicle sensors measure sound and / or vibrations associated with the surface elements. The surface elements are then identified based on patterns in the sensor measurements. Based on the road surface, the vehicle can determine an appropriate vehicle response to encountering the surface element, which allows the vehicle to better respond to the surface element. Additionally, the vehicle can compare the measurements of the surface element to historical measurements of the surface element. Doing so allows the vehicle to determine a calibration state of the sensors that performed the historical measurements. In some examples, the vehicle publishes the identified surface elements and associated sensor measurements to a shared database.

[0029] Improvements in the behavior of the vehicle based on surface element adaptation improve the safety and reliability of the vehicle. Additionally, the use of sound and / or vibration measurements to detect surface elements enables the vehicle to detect scenarios and conditions that would otherwise go undetected using other types of measurements. Moreover, comparing the measurements to historical measurements helps mitigate sensor failure and / or detect sensor damage. Publishing the identified road elements to a shared database allows vehicle planning systems to leverage historical information to prepare for known road disturbances.

[0030] System Overview

[0031] Figure 1 An example of an autonomous vehicle 100 with autonomous capabilities is shown.

[0032] As used herein, the term“autonomous capabilities” refers to a function, feature, or facility that enables a vehicle to operate, in part or in whole, without real-time human intervention, including but not limited to fully autonomous vehicles, highly autonomous vehicles, and conditional autonomous vehicles.

[0033] As used herein, an autonomous vehicle (AV) is a vehicle with autonomous capabilities.

[0034] As used herein, a“vehicle” includes a means of transporting goods or people. For example, a car, a bus, a train, an airplane, a drone, a truck, a boat, a ship, a submersible, a spacecraft, etc. A self-driving car is an example of a vehicle.

[0035] As used herein, a“trajectory” refers to a path or route that navigates an AV from a first spatiotemporal location to a second spatiotemporal location. In embodiments, the first spatiotemporal location is referred to as an initial or starting location, and the second spatiotemporal location is referred to as a destination, final location, target, target location, or target location. In some examples, a trajectory is composed of one or more road segments (e.g., segments of a road), and each road segment is composed of one or more blocks (e.g., a portion of a lane or an intersection). In embodiments, the spatiotemporal locations correspond to real-world locations. For example, a spatiotemporal location is a pickup or drop-off location for a person or cargo.

[0036] As used herein,“sensor(s)” include one or more hardware components for detecting information about the environment surrounding the sensor. Some hardware components can include sensing components (e.g., image sensors, biometric sensors), transmitting and / or receiving components (e.g., laser or radio frequency wave emitters and receivers), electronic components such as analog-to-digital converters, data storage devices such as RAM and / or non-volatile memory, software or firmware components, and data processing components such as application-specific integrated circuits, microprocessors, and / or microcontrollers.

[0037] As used herein, a “scene description” is a data structure (e.g., a list) or data stream that includes one or more classified or labeled objects detected by one or more sensors on an AV vehicle, or one or more classified or labeled objects provided by a source external to the AV.

[0038] As used herein, a "road" is a physical area that can be traversed by a vehicle and can correspond to a named thoroughfare (e.g., a city street, an interstate highway, etc.) or can correspond to an unnamed thoroughfare (e.g., a driveway within a house or office building, a section of a parking lot, a section of a vacant parking lot, a dirt road in a rural area, etc.). Because some vehicles (e.g., four-wheel drive pickup trucks, off-road vehicles (SUVs), etc.) can traverse a variety of physical areas that are not particularly suitable for vehicle travel, a "road" can be any physical area that is not formally defined as a thoroughfare by a municipality or other governmental or administrative agency.

[0039] As used herein, a “lane” is a portion of a road that can be traversed by a vehicle. Lanes are sometimes identified based on lane markings. For example, a lane may correspond to most or all of the space between lane markings, or only a portion of the space between lane markings (e.g., less than 50%). For example, a road with lane markings that are far apart may accommodate two or more vehicles, allowing one vehicle to pass another without crossing the lane markings, and thus may be interpreted as a lane narrower than the space between lane markings, or as having two lanes between lanes. Lanes may also be interpreted in the absence of lane markings. For example, lanes may be defined based on physical features of the environment (e.g., rocks and trees along an avenue in a rural area, or natural obstacles to be avoided, for example, in less developed areas). Lanes may also be interpreted independently of lane markings or physical features. For example, a lane may be interpreted based on an arbitrary path through an area without obstacles that would otherwise lack features that would be interpreted as lane boundaries. In an example scenario, an AV may interpret a lane as passing through an obstacle-free portion of a field or open space. In another example scenario, an AV may interpret a lane through a wide (e.g., wide enough for two or more lanes) road that does not have lane markings. In this scenario, the AV may communicate lane-related information to other AVs so that the other AVs can use the same lane information to coordinate path planning between the AVs.

[0040] The term “Over-the-Air (OTA) Client” includes any AV, or any electronic device (e.g., a computer, controller, IoT device, Electronic Control Unit (ECU)) embedded in, coupled to, or communicating with an AV.

[0041] The term "over-the-air (OTA) update" means any update, change, deletion, or addition to software, firmware, data, or configuration settings delivered to an OTA client using proprietary and / or standardized wireless communication technologies including, but not limited to: cellular mobile communication (e.g., 2G, 3G, 4G, 5G), wireless radio area networks (e.g., WiFi), and / or satellite Internet.

[0042] The term "edge node" refers to one or more edge devices coupled to a network that provide a portal for communication with AVs and can communicate with other edge nodes and cloud-based computing platforms to schedule and deliver OTA updates to OTA clients.

[0043] The term "edge device" refers to a device that implements an edge node and provides a physical wireless access point (AP) to an enterprise or service provider (e.g., VERIZON, AT&T) core network. Examples of edge devices include, but are not limited to: computers, controllers, transmitters, routers, routing switches, integrated access devices (IADs), multiplexers, metropolitan area network (MAN) and wide area network (WAN) access devices.

[0044] "one or more" includes a function performed by one element, a function performed by more than one element, e.g., in a distributed manner, a function performed by one element in a number of instances, a function performed by more than one element in a number of instances, or any combination of the above.

[0045] It will also be understood that, although the terms "first," "second," etc. can be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first contact could be termed a second contact, and, similarly, a second contact could be termed a first contact, without departing from the scope of the various described embodiments. The first contact and the second contact are both contacts, but they are not the same contact.

[0046] The terminology used in the description of the various embodiments described herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used in the description of the various embodiments and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term “and / or” as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items. It will be further understood that the terms “comprises,” “comprising,” “includes,” “including,” “has,” “having,” and / or “has” when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0047] As used herein, the term “if’ can be construed to mean “when” or “when” or “in response to a determination” or “in response to a detection” depending on the context. Similarly, the phrase “if it is determined” or “if [a stated condition or event] has been detected” can be construed to mean “upon a determination” or “in response to a determination” or “upon detecting [a stated condition or event]” or “in response to detecting [a stated condition or event]”, depending on the context.

[0048] As used herein, an AV system refers to an array of AVs and hardware, software, stored data, and real-time generated data that support operation of the AVs. In embodiments, the AV system is incorporated within the AVs. In embodiments, the AV system is distributed across several locations. For example, some software of the AV system is implemented on a cloud computing environment similar to the cloud computing environment 200 described below with respect to FIG. 2. Figure 2 The cloud computing environment described below with respect to FIG. 2.

[0049] In general, this document describes technology applicable to any vehicle with one or more levels of autonomy, including fully autonomous vehicles, highly autonomous vehicles, and conditionally autonomous vehicles, such as so-called Level 5, Level 4, and Level 3 vehicles, respectively (see SAE International Standard J3016: Taxonomy and Definitions for Terms Related to Driving Automation Systems for Road Vehicles, incorporated by reference in its entirety for more detailed information on levels of vehicle autonomy). The technology described in this document is also applicable to partially autonomous vehicles and driver-assist vehicles, such as so-called Level 2 and Level 1 vehicles (see SAE International Standard J3016: Taxonomy and Definitions for Terms Related to Driving Automation Systems for Road Vehicles). In embodiments, one or more Level 1, Level 2, Level 3, Level 4, and Level 5 vehicle systems can automatically perform certain vehicle operations (e.g., steering, braking, and use of a map) under certain operating conditions based on processing of sensor inputs. The technology described in this document can benefit vehicles at any level, ranging from fully autonomous vehicles to human-operated vehicles.

[0050] Autonomous vehicles have advantages over vehicles that require a human driver. One advantage is safety. For example, in 2016, the United States experienced 6 million car accidents, 2.4 million people injured, 40,000 people killed, and 13 million vehicle crash incidents, with an estimated societal cost of $910 billion. From 1965 to 2015, the number of traffic fatalities per 100 million miles traveled in the United States has decreased from about 6 to about 1, in part due to additional safety measures deployed in vehicles. For example, warnings of an additional half-second associated with an impending collision are believed to mitigate 60% of rear-end collisions. However, passive safety features (e.g., seat belts, airbags) can have reached their limit in improving this number. Thus, active safety measures such as automated control of a vehicle are a possible next step in improving these statistics. Since a human driver is believed to be the cause of a serious pre-crash event in 95% of collisions, automated driving systems can achieve better safety outcomes by, for example: reliably identifying and avoiding emergencies better than a human; making better decisions than a human, following traffic laws better than a human, and predicting future events better than a human; and reliably controlling a vehicle better than a human.

[0051] Reference Figure 1 The AV system 120 causes the vehicle 100 to operate along a trajectory 198 through the environment 190 to a destination 199 (sometimes referred to as a final location) while avoiding objects (e.g., natural obstacles 191, vehicles 193, pedestrians 192, cyclists, and other obstacles) and obeying road rules (e.g., operating rules or driving preferences).

[0052] In an embodiment, the AV system 120 includes a device 101 for receiving and operating operational commands from a computer processor 146. The term "operational command" is used to refer to an executable instruction (or set of instructions) that causes a vehicle to perform an action (e.g., a driving maneuver). The operational commands may include, but are not limited to, instructions for causing the vehicle to start moving forward, stop moving forward, start moving backward, stop moving backward, accelerate, decelerate, make a left turn, and make a right turn. In an embodiment, the computer processor 146 and the following reference Figure 3 The processor 304 is similarly described. Examples of devices 101 include steering controls 102, brakes 103, gears, an accelerator pedal or other acceleration control mechanism, windshield wipers, side door locks, window controls, and turn indicators.

[0053] In an embodiment, the AV system 120 includes sensors 121 for measuring or inferring attributes of the state or condition of the vehicle 100, such as the AV's position, linear and angular velocity and acceleration, and heading (e.g., the direction of the front end of the vehicle 100). Examples of sensors 121 are GPS, an inertial measurement unit (IMU) that measures both linear acceleration and angular rate of the vehicle, wheel rate sensors for measuring or estimating wheel slip, wheel brake pressure or brake torque sensors, engine torque or wheel torque sensors, and steering angle and angular rate sensors.

[0054] In an embodiment, the sensors 121 also include sensors for sensing or measuring properties of the AV's environment, such as a monocular or stereo camera 122 in the visible, infrared, or thermal (or both) spectrum, a LiDAR 123, a RADAR, an ultrasonic sensor, a time-of-flight (TOF) depth sensor, a velocity sensor, a temperature sensor, a humidity sensor, and a precipitation sensor.

[0055] In an embodiment, the AV system 120 includes a data storage unit 142 and a memory 144 for storing machine instructions associated with the computer processor 146 or data collected by the sensor 121. In an embodiment, the data storage unit 142 is associated with the following Figure 3 ROM 308 or storage device 310 described above. In an embodiment, memory 144 is similar to main memory 306 described below. In an embodiment, data storage unit 142 and memory 144 store historical, real-time, and / or predictive information about environment 190. In an embodiment, the stored information includes maps, driving performance, traffic congestion updates, or weather conditions. In an embodiment, data related to environment 190 is transmitted from remote database 134 to vehicle 100 via a communication channel.

[0056] In embodiments, the AV system 120 includes communication devices 140 for transmitting properties of other vehicles' states and conditions, such as position, linear and angular velocity, linear and angular acceleration, and linear and angular heading, measured or inferred to the vehicle 100. These devices include vehicle-to-vehicle (V2V) and vehicle-to-infrastructure (V2I) communication devices and devices for wireless communication over point-to-point or ad hoc networks or both. In embodiments, the communication devices 140 communicate across the electromagnetic spectrum, including radio and optical communications, or other media (e.g., air and acoustic media). The combination of vehicle-to-vehicle (V2V), vehicle-to-infrastructure (V2I) communication (and, in some embodiments, one or more other types of communication) is sometimes referred to as vehicle-to-everything (V2X) communication. V2X communication is generally in compliance with one or more communication standards for communication by and between autonomous vehicles.

[0057] In embodiments, the communication devices 140 include communication interfaces. For example, wired, wireless, WiMAX, Wi-Fi, Bluetooth, satellite, cellular, optical, near field, infrared, or radio interfaces. The communication interfaces transfer data from the remote database 134 to the AV system 120. In embodiments, the remote database 134 is embedded in a cloud computing environment 200 as described in Figure 2 In embodiments, the communication devices 140 transfer data collected from the sensors 121 or other data related to the operation of the vehicle 100 to the remote database 134. In embodiments, the communication devices 140 transfer information related to teleoperation to the vehicle 100. In some embodiments, the vehicle 100 communicates with other remote (e.g., "cloud") servers 136.

[0058] In embodiments, the remote database 134 also stores and transfers digital data (e.g., storing data such as road and street locations). This data is stored in the memory 144 on the vehicle 100 or transferred from the remote database 134 to the vehicle 100 over a communication channel.

[0059] In embodiments, the remote database 134 stores and transfers historical information related to the driving properties (e.g., velocity and acceleration profiles) of vehicles that have previously traveled along the trajectory 198 at similar times of day. In one implementation, this data can be stored in the memory 144 on the vehicle 100 or transferred from the remote database 134 to the vehicle 100 over a communication channel.

[0060] The computer processor 146 located on the vehicle 100 generates control actions algorithmically based on both real-time sensor data and a priori information, allowing the AV system 120 to perform its autonomous driving capabilities.

[0061] In embodiments, the AV system 120 includes a computer peripheral 132 coupled to the computer processor 146 for providing information and reminders to a user of the vehicle 100 (e.g., an occupant or a remote user) and receiving input from the user. In embodiments, the peripheral 132 is similar to the display 312, input device 314, and cursor controller 316 discussed below with reference to FIG. 3. The coupling is wireless or wired. Any two or more of the interface devices can be integrated into a single device. Figure 3

[0062] In embodiments, the AV system 120 receives and enforces a privacy level of the occupant, e.g., specified by the occupant or stored in a profile associated with the occupant. The privacy level of the occupant determines how the specific information associated with the occupant (e.g., occupant comfort data, biometric data, etc.) stored in the occupant profile and / or stored on the cloud server 136 and associated with the occupant profile is permitted to be used. In embodiments, the privacy level specifies specific information associated with the occupant that is deleted upon completion of the ride. In embodiments, the privacy level specifies specific information associated with the occupant and identifies one or more entities that are authorized to access the information. Examples of the specified entities that are authorized to access the information can include other AVs, third-party AV systems, or any entity that can potentially access the information.

[0063] The privacy level of the occupant can be specified at one or more levels of granularity. In embodiments, the privacy level identifies specific information to be stored or shared. In embodiments, the privacy level applies to all information associated with the occupant, such that the occupant can specify that her personal information is not stored or shared. The specification of entities that are permitted to access specific information can also be specified at various levels of granularity. Various sets of entities that are permitted to access specific information can include, for example, other AVs, the cloud server 136, specific third-party AV systems, etc.

[0064] In embodiments, the AV system 120 or the cloud server 136 determines whether the AV 100 or another entity has access to certain information associated with the occupant. For example, a third-party AV system attempting to access occupant input related to a specific spatiotemporal location must obtain authorization, e.g., from the AV system 120 or the cloud server 136, to access information associated with the occupant. For example, the AV system 120 uses the specified privacy level of the occupant to determine whether the occupant input related to the spatiotemporal location can be presented to the third-party AV system, the AV 100, or another AV. This enables the privacy level of the occupant to specify which other entities are permitted to receive data related to the actions of the occupant or other data associated with the occupant.

[0065] Figure 2 ​An example "cloud" computing environment is shown. Cloud computing is a service delivery model for enabling convenient, on-demand access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) over a network. In a typical cloud computing system, one or more large cloud data centers house the machines used to deliver the services provided by the cloud. Now referring to Figure 2 , cloud computing environment 200 includes cloud data centers 204a, 204b, and 204c interconnected by a cloud 202. Data centers 204a, 204b, and 204c provide cloud computing services to computer systems 206a, 206b, 206c, 206d, 206e, and 206f connected to cloud 202.

[0066] The cloud computing environment 200 includes one or more cloud data centers. Generally speaking, a cloud data center (e.g. Figure 2 The cloud data center 204a shown in FIG refers to a cloud (eg Figure 2 The physical arrangement of servers in a cloud 202 (or a specific portion of a cloud) as shown in FIG. For example, servers are physically arranged into rooms, groups, rows, and racks in a cloud data center. A cloud data center has one or more zones, which include one or more server rooms. Each room has one or more rows of servers, and each row includes one or more racks. Each rack includes one or more individual server nodes. In some implementations, servers in zones, rooms, racks, and / or rows are arranged into groups based on the physical infrastructure requirements of the data center facility, including power, energy, heat, heat sources, and / or other requirements. In an embodiment, the server nodes are similar to Figure 3 The data center 204a has many computing systems distributed across multiple racks.

[0067] Cloud 202 includes cloud data centers 204a, 204b, and 204c, as well as networks and network resources (e.g., network devices, nodes, routers, switches, and network cables) used to connect cloud data centers 204a, 204b, and 204c and facilitate access to cloud computing services by computing systems 206a-f. In embodiments, the network represents any combination of one or more local networks, wide area networks, or internetworks coupled via wired or wireless links deployed using terrestrial or satellite connections. Data exchanged over the network is transmitted using a variety of network layer protocols, such as Internet Protocol (IP), Multiprotocol Label Switching (MPLS), Asynchronous Transfer Mode (ATM), Frame Relay, and the like. Furthermore, in embodiments where the network represents a combination of multiple subnetworks, a different network layer protocol is used on each underlying subnetwork. In some embodiments, the network represents one or more interconnected internetworks, such as the public Internet.

[0068] The computing systems 206a-f or cloud computing service consumers connect to the cloud 202 through network links and network adapters. In embodiments, the computing systems 206a-f are implemented as various computing devices, such as servers, desktops, laptops, tablets, smartphones, Internet of Things (IoT) devices, autonomous vehicles (including cars, drones, spacecraft, trains, buses, etc.), and consumer electronics. In embodiments, the computing systems 206a-f are implemented in or as part of other systems.

[0069] Figure 3 A computer system 300 is shown. In implementations, the computer system 300 is a special-purpose computing device. The special-purpose computing device is either hard-wired to perform the techniques, or includes digital electronic devices such as one or more application-specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs) that are persistently programmed to perform the techniques, or includes one or more general purpose hardware processors programmed to perform the techniques pursuant to program instructions in firmware, memory, other storage, or a combination. Such special-purpose computing devices can also combine custom hard-wired logic, ASICs, or FPGAs with custom programming to accomplish the techniques. In various embodiments, the special-purpose computing device is a desktop computer system, portable computer system, handheld device, network device, or any other device that incorporates hard-wired and / or program logic to implement the techniques.

[0070] In embodiments, the computer system 300 includes a bus 302 or other communication mechanism for communicating information, and a processor 304 coupled to the bus 302 for processing information. The processor 304 is, for example, a general-purpose microprocessor. The computer system 300 also includes a main memory 306, such as a random access memory (RAM) or other dynamic storage device, coupled to the bus 302 for storing information and instructions to be executed by the processor 304. In one implementation, the main memory 306 is used for storing temporary variables or other intermediate information during execution of instructions to be executed by the processor 304. The computer system 300 can further include a read only memory (ROM) 308 or other static storage device coupled to the bus 302 for storing static information and instructions for the processor 304. A storage device 310, such as a magnetic disk, optical disk, solid-state drive, or three-dimensional crosspoint memory, is provided and coupled to the bus 302 for storing information and instructions.

[0071] In embodiments, the computer system 300 also includes a read only memory (ROM) 308 or other static storage device coupled to the bus 302 for storing static information and instructions for the processor 304. A storage device 310, such as a magnetic disk, optical disk, solid-state drive, or three-dimensional crosspoint memory, is provided and coupled to the bus 302 for storing information and instructions.

[0072] In embodiments, computer system 300 is coupled via bus 302 to a display 312, such as a cathode ray tube (CRT), liquid crystal display (LCD), plasma display, light-emitting diode (LED) display, or organic light-emitting diode (OLED) display for displaying information to a computer user. An input device 314, including alphanumeric and other keys, is coupled to bus 302 for communicating information and command selections to processor 304. Another type of user input device is cursor control 316, such as a mouse, a trackball, a touchscreen, or cursor direction keys for communicating direction information and command selections to processor 304 and for

[0073] According to one embodiment, the techniques herein are performed by computer system 300 in response to processor 304 executing one or more sequences of one or more instructions contained in main memory 306. Such instructions can be read into main memory 306 from another storage medium, such as storage device 310. Execution of the sequences of instructions contained in main memory 306 causes processor 304 to perform the process steps described herein. In alternative embodiments, hard-wired circuitry can be used in place of or in combination with software instructions.

[0074] The term “storage media” as used herein refers to any non-transitory media that store data and / or instructions that cause a machine to operate in a specific fashion. Such storage media include non-volatile media and / or volatile media. Non-volatile media include, for example, optical disks, magnetic disks, solid-state drives, or three-dimensional cross-point memory such as storage device 310. Volatile media include dynamic memory, such as main memory 306. Common forms of storage media include, for example, a floppy disk, a flexible disk, a hard disk, a solid- state drive, magnetic tape, or any other magnetic data storage medium, a CD-ROM, any other optical data storage medium, any physical medium with patterns of holes, a RAM, a PROM, and EPROM, a FLASH-EPROM, an NV-RAM, or any other memory chip or cartridge.

[0075] Storage media are distinct from, but can be used in combination with, transmission media. Transmission media participate in transferring information between storage media. For example, transmission media include coaxial cables, copper wire, and fiber optic cables, including wires that provide bus 302. Transmission media can also take the form of acoustic or light waves, such as those generated during radio frequency and infrared data communications.

[0076] In one embodiment, various forms of media are involved in carrying one or more sequences of one or more instructions to processor 304 for execution. For example, the instructions may initially be executed on a disk or solid-state drive of a remote computer. The remote computer loads the instructions into its dynamic memory and sends the instructions over a telephone line using a modem. The modem local to computer system 300 receives the data on the telephone line and uses an infrared transmitter to convert the data to an infrared signal. An infrared detector receives the data carried in the infrared signal, and appropriate circuitry places the data on bus 302. Bus 302 carries the data to main memory 306, from which processor 304 retrieves and executes the instructions. The instructions received by main memory 306 may optionally be stored on storage device 310 before or after execution by processor 304.

[0077] Computer system 300 also includes a communication interface 318 coupled to bus 302. Communication interface 318 provides a two-way data communication coupled to a network link 320 that is connected to a local network 322. For example, communication interface 318 is an integrated services digital network (ISDN) card, a cable modem, a satellite modem, or a modem for providing a data communication connection with a corresponding type of telephone line. As another example, communication interface 318 is a local area network (LAN) card for providing a data communication connection with a compatible LAN. In some implementations, a wireless link is also implemented. In any such implementation, communication interface 318 sends and receives electrical, electromagnetic, or optical signals that carry digital data streams representing various types of information.

[0078] Network link 320 typically provides data communication to other data devices through one or more networks. For example, network link 320 provides a connection to a host computer 324 or to a cloud data center or facility operated by an Internet Service Provider (ISP) 326 through a local network 322. ISP 326, in turn, provides data communication services through the worldwide packet data communication network now commonly referred to as the "Internet" 328. Both local network 322 and Internet 328 use electrical, electromagnetic, or optical signals that carry digital data streams. The signals through the various networks and the signals on network link 320 and through communication interface 318 are example forms of transmission media, where these signals carry digital data to and from computer system 300. In an embodiment, network 320 includes cloud 202 or a portion of cloud 202 as described above.

[0079] Computer system 300 sends messages and receives data, including program code, through network(s), network link 320, and communication interface 318. In an embodiment, computer system 300 receives code for processing. The received code is executed by processor 304 upon receipt and / or stored in storage device 310 or other non-volatile storage for later execution.

[0080] Autonomous Vehicle Architecture

[0081] Figure 4 Shown for autonomous vehicles (e.g., Figure 1 100). The architecture 400 includes a perception module 402 (sometimes referred to as perception circuitry), a planning module 404 (sometimes referred to as planning circuitry), a control module 406 (sometimes referred to as control circuitry), a positioning module 408 (sometimes referred to as positioning circuitry), and a database module 410 (sometimes referred to as database circuitry). Each module plays a role in the operation of the vehicle 100. Collectively, the modules 402, 404, 406, 408, and 410 may be Figure 1 406, 408, and 410 are each sometimes referred to as a processing circuit (e.g., computer hardware, computer software, or a combination thereof). A combination of any or all of modules 402, 404, 406, 408, and 410 is also an example of a processing circuit.

[0082] In use, planning module 404 receives data representing destination 412 and determines data representing a trajectory 414 (sometimes referred to as a route) that vehicle 100 may travel in order to reach (e.g., arrive at) destination 412. In order for planning module 404 to determine data representing trajectory 414, planning module 404 receives data from perception module 402, positioning module 408, and database module 410.

[0083] The perception module 402 uses, for example, Figure 1 One or more sensors 121 are shown to identify nearby physical objects, classify the objects (e.g., into types such as pedestrians, bicycles, cars, traffic signs, etc.), and provide a scene description including the classified objects 416 to the planning module 404.

[0084] The planning module 404 also receives data representing the AV position 418 from the positioning module 408. The positioning module 408 determines the AV position by calculating the position using data from the sensor 121 and data from the database module 410 (e.g., geographic data). For example, the positioning module 408 uses data from a GNSS (Global Navigation Satellite System) sensor and geographic data to calculate the longitude and latitude of the AV. In an embodiment, the data used by the positioning module 408 includes a high-precision map with lane geometry, a map describing the road network connection properties, a map describing the physical properties of the lanes (such as traffic speed, traffic volume, the number of vehicle and bicycle lanes, lane width, lane traffic direction, or lane marking type and location, or a combination thereof), and a map describing the spatial location of road elements (such as intersections, traffic signs, or various types of other driving signals). In an embodiment, the high-precision map is constructed by adding data to the low-precision map via automatic or manual annotation.

[0085] The control module 406 receives data representing the trajectory 414 and data representing the AV's position 418 and operates the AV's control functions 420a-420c (e.g., steering, throttle, brakes, ignition) in a manner that will cause the vehicle 100 to travel the trajectory 414 to reach the destination 412. For example, if the trajectory 414 includes a left turn, the control module 406 will operate the control functions 420a-420c in such a manner that the steering angle of the steering function will cause the vehicle 100 to turn left, and the throttle and brakes will cause the vehicle 100 to pause and wait for a passing pedestrian or vehicle before making the turn.

[0086] Path Planning

[0087] Figure 5 Shown (for example, Figure 4 5. A block diagram 500 of the relationship between the inputs and outputs of the planning module 404 (shown in FIG. 5 ) is shown. Generally speaking, the output of the planning module 404 is a route 502 from a starting point 504 (e.g., a source location or initial location) to an end point 506 (e.g., a destination or final location). The route 502 is typically defined by one or more road segments. For example, a road segment refers to a distance to be traveled on at least a portion of a street, road, highway, lane, or other physical area suitable for automobile travel. In some examples, for example, if the vehicle 100 is an off-road capable vehicle such as a four-wheel drive (4WD) or all-wheel drive (AWD) car, SUV, or pickup truck, the route 502 includes "off-road" sections such as unpaved paths or open fields.

[0088] In addition to route 502, the planning module also outputs lane-level route planning data 508. Lane-level route planning data 508 is used to drive through the road segments of route 502 at a specific time based on the conditions of the road segments. For example, if route 502 includes a multi-lane highway, the lane-level route planning data 508 includes trajectory planning data 510, wherein the vehicle 100 can use the trajectory planning data 510 to select a lane from the multiple lanes based on, for example, whether an exit is approaching, whether there are other vehicles in one or more of the multiple lanes, or other factors that change over the course of a few minutes or less. Similarly, in some implementations, the lane-level route planning data 508 includes a rate constraint 512 specific to a road segment of route 502. For example, if the road segment includes pedestrians or unexpected traffic, the rate constraint 512 can limit the vehicle 100 to a travel speed slower than the expected speed, such as a speed based on the speed limit data for the road segment.

[0089] In an embodiment, inputs to the planning module 404 include (e.g., Figure 4 ) database data 514, current location data 516 (e.g., Figure 4 AV position 418 shown), (e.g., for Figure 4 Destination data 518 and object data (e.g., Figure 4 4. The classification of objects 416 perceived by the perception module 402 is shown in FIG. 4. In some embodiments, the database data 514 includes rules used in planning. The rules are specified using a formal language (e.g., using Boolean logic). In any given situation encountered by the vehicle 100, at least some of these rules will apply to that situation. A rule applies to a given situation if it has conditions that are satisfied based on information available to the vehicle 100 (e.g., information about the surrounding environment). Rules can have priorities. For example, a rule that says "if the road is a freeway, move to the far left lane" can have a lower priority than a rule that says "if the exit is within one mile, move to the far right lane."

[0090] Autonomous vehicle control

[0091] Figure 6 Shown (for example, Figure 4A block diagram 600 of inputs and outputs of the control module 406 (shown) is shown. The control module operates in accordance with a controller 602 that includes, for example: one or more processors (e.g., one or more computer processors such as microprocessors or microcontrollers or both) similar to the processor 304; short-term and / or long-term data storage (e.g., memory, random access memory or flash memory or both) similar to the main memory 306, ROM 308, and storage 310; and instructions stored in memory that, when executed (e.g., by the one or more processors), perform the operations of the controller 602.

[0092] In embodiments, the controller 602 receives data representing a desired output 604. The desired output 604 typically includes velocity, such as speed and heading. The desired output 604 can be based, for example, on data received from (e.g., as described above with respect to) the planning module 404 (shown). Figure 4 The controller 602 generates data that can be used as a throttle input 606 and a steering input 608 in accordance with the desired output 604. The throttle input 606 represents, for example, the size of the throttle (e.g., acceleration control) of the vehicle 100 that should be engaged to achieve the desired output 604, such as by engaging a throttle pedal or engaging another throttle control. In some examples, the throttle input 606 also includes data that can be used to engage a brake (e.g., deceleration control) of the vehicle 100. The steering input 608 represents the angle at which the steering control (e.g., steering wheel, steering angle actuator, or other function used to control the steering angle) of the AV should be positioned to achieve the desired output 604.

[0093] In embodiments, the controller 602 receives feedback that is used in adjusting the inputs provided to the throttle and steering. For example, if the vehicle 100 encounters an interference 610, such as a hill, the measured speed 612 of the vehicle 100 drops below the desired output speed. In embodiments, any measured outputs 614 are provided to the controller 602 so that the necessary adjustments are made, for example, based on the difference 613 between the measured speed and the desired output. The measured outputs 614 include measured position 616, measured speed 618 (including speed and heading), measured acceleration 620, and other outputs that are measurable by sensors of the vehicle 100.

[0094] In embodiments, information about the interference 610 is detected in advance, for example, by sensors such as cameras or LiDAR sensors, and the information is provided to a predictive feedback module 622. The predictive feedback module 622 then provides information to the controller 602 that the controller 602 can use to adjust accordingly. For example, if a sensor of the vehicle 100 detects (“sees”) a hill, the controller 602 can use that information to prepare to engage the throttle at the appropriate time to avoid a significant deceleration.

[0095] Surface-directed decision making

[0096] Figure 7 An example system 700 for surface-directed decision making is illustrated. The system 700 is configured to identify surface elements along a path of a vehicle. The system 700 is also configured to adapt behavior of the vehicle based on the identified surface elements. Surface elements are any raised surfaces or surface depressions that the vehicle can contact. Example surface elements include raised pavement markers (e.g., reflective / non-reflective pavement markers), guardrails, drain grates, road dividers, painted road markings, parking blocks / chalk marks / bumpers, rumble strips, speed bumps, sidewalks, curbs, road connections, expansion or moving joints, potholes, ruts, and grooves.

[0097] As shown, the system 700 includes sensors 702 (e.g., the same or similar sensors as the sensors 121 of FIG. 1), a signal processor 704 (e.g., the same or similar processor as the processor 146 of FIG. 1 or the processor 304 of FIG. 3), a behavior selector 706, a motion planner 708 (e.g., the same or similar motion planner as the planning module 404 of FIG. 4 and as described in FIG. 5), and a controller 710 (e.g., the same or similar controller as the control module 406 of FIG. 4 and / or the controller 602 of FIG. 6). Figure 7 Figure 1 Figure 1 Figure 3 Figure 4 Figure 5 Figure 4 Figure 6

[0098] In some examples, one or more components of the system 700 are implemented using the same or similar computer system as the computer system 300 described in FIG. 3. Additionally or alternatively, one or more components can be implemented on the same or similar cloud computing environment as the cloud computing environment 200 described in FIG. 2. Note that the system 700 is shown for illustrative purposes only, as the system 700 can include additional components and / or remove one or more components without departing from the scope of the present disclosure. Moreover, the components of the system 700 can be arranged and connected in any manner. Although the following discussion describes the system 700 in the context of identifying one surface element along a path of a vehicle, the system 700 can identify more than one surface element simultaneously or consecutively. Figure 3 Figure 2

[0099] ​​​​​​​​​​In embodiments, the sensors 702 include at least one sensor configured to capture data associated with a surface element along a path of a vehicle. The sensors 702 are also referred to as a surface sensor suite. Example sensors include a vibration sensor, a tilt sensor, a shock sensor, an accelerometer, a gyroscope, a tire pressure sensor, a suspension pressure sensor, an extensometer, a sound pressure sensor, and a microphone, among others. The sensors 702 are arranged on the vehicle in one or more arrangements, such as an array arrangement, a one-sensor-per-wheel arrangement, or a center arrangement, among others. In an array arrangement, multiple sensors are arranged in rows and columns across a region of the vehicle. In a one-sensor-per-wheel arrangement, a respective sensor is disposed on at least one wheel of the vehicle. In a center arrangement, at least one sensor is located at a center of the vehicle. In examples where the sensors 702 include more than one type of sensor, different types of sensors can include different numbers and different arrangements.

[0100] In embodiments, the sensors 702 are configured to measure data indicative of sound and / or vibrations associated with a surface element. That is, the sensors 702 measure sound and / or vibrations caused by the surface element. In some examples, the sensors 702 measure the sound and / or vibrations when the vehicle contacts the surface element (e.g., when a tire contacts the surface element). In other examples, the sensors 702 measure the sound and / or vibrations when other vehicles contact the surface element. The sensors 702 provide sensor data to the signal processor 704.

[0101] In embodiments, the signal processor 704 is configured to perform operations related to processing the sensor data. Example operations include pre-processing the data to remove noise from the data and extracting patterns from the data that can be used to identify the surface element. More specifically, the data is pre-processed to filter out noise (e.g., using a low-pass filter) or to isolate the impact of the surface element from other known factors that impact the sensor signal (e.g., changes in the direction of gravity relative to road grade). In one example, machine learning is used to train the signal processor 704 to extract patterns to identify the surface element. In this example, data is collected, annotated to correspond to one of the specified classes of surface elements, and then a neural network is trained, for example, to perform classification. In one implementation, the neural network is a recurrent neural network. In this implementation, raw sensor data is fed into the network with some fixed look-back time window for recursion. In other implementations, a non-recurrent network (e.g., an artificial neural network, ANN) is used. For example, a Fourier transform is performed over a time window of the sensor data (e.g., the amplitude of the signal is greater than a predetermined frequency range). The transformed data is provided as input features to the neural network.

[0102] In other examples, the signal processor 704 is trained using a model-based approach. In this example, the signals from the sensors are related to equations based on guided physics, and the surface elements are structured to have a high degree of regularity. More specifically, if the car suspension parameters are known and the encountered speed bumps are all standard size, a formula is derived for the expected vertical acceleration at each location on the car body as a function of the forward car speed when crossing the speed bump. Thus, a vertical acceleration sensor signal within a predetermined threshold of the expected value is treated as an observation of a speed bump.

[0103] In one example, the signal processor 704 is trained to recognize and extract such data patterns using known data patterns. The known data patterns can be expected frequency variations in the data. For example, when driving over rumble strips, the data pattern is an expected frequency in the vertical acceleration signal (e.g., as a function of the driving speed for a given single strip separation distance). The known data patterns can also be amplitude and direction expectations. For example, the amplitude and direction of the signal can be used to distinguish a pothole or depression from a speed bump. More specifically, driving over a speed bump will expect some rate-dependent time delay between the vertical rise and the following drop, which corresponds to the standard width of a speed bump. In addition, the amplitude of the acceleration signal expected from a curb impact will be much greater than the amplitude of the acceleration signal experienced on most speed bumps.

[0104] In other examples, the signal processor 704 compares the sensor data to known data patterns associated with known surface elements. In this example, if the signal processor 704 identifies a threshold similarity between the pattern in the sensor data and the known data pattern, the signal processor 704 identifies the surface element as the surface element associated with the known data pattern. The threshold similarity is a measure of similarity between the sensor data pattern and the known data pattern that is greater than a predetermined threshold (e.g., greater than 90% similarity between the sensor data pattern and the known data pattern). The signal processor 704 provides the identification of the surface element to the behavior selector 706.

[0105] In embodiments, the behavior selector 706 determines a vehicle behavior based on the identified surface element. The vehicle behavior includes a vehicle trajectory and / or a vehicle driving setting (e.g., vehicle speed; torque at a motor, wheel, etc.; acceleration and / or suspension mode). In implementations, the behavior selector 706 selects a vehicle behavior associated with the identified surface element from a database of known vehicle behaviors. The database of known vehicle behaviors can include a lookup table or a decision tree. As an example, when the identified surface element is a parking block, the behavior selector 706 selects a vehicle behavior that pauses a parking maneuver, which is a vehicle behavior associated with a parking block. As other examples, when the identified surface element (e.g., a lane marking, rumble strip, or road marking) indicates that the vehicle has left its lane, adjusting the steering to return to the center of the lane is an action that is traceable to a known desired behavior of staying within the lane.

[0106] In embodiments, the behavior selector 706 does not determine a vehicle behavior directly from the identified surface element, but first determines that the vehicle is encountering a particular scenario based on the identified surface element. The behavior selector 706 then selects a vehicle behavior based on the particular scenario. In an example where the identified surface element is an off-road surface (e.g., rock or sand), the behavior selector 706 determines that at least one wheel is off-road. In response, the behavior selector 706 determines to limit the vehicle speed and / or steering wheel angle. Additionally, in an example where the identified road element is a curb, the behavior selector 706 determines that at least one front wheel of the vehicle is blocked. In response, the behavior selector 706 determines that a reverse maneuver is necessary and should stop any forward motion currently.

[0107] In embodiments, the behavior selector 706 adds a representation of the identified surface element to a shared database that can be accessed by a fleet of vehicles. The representation includes a location of the surface element, sensor data associated with the surface element, a data pattern identified in the sensor data, and / or a selected vehicle behavior associated with the surface element. In some examples, the behavior selector 706 obtains expected road elements along a vehicle path from the shared database. The expected road elements are historical road elements that were previously detected along the vehicle path, or there is information indicating that the road elements are located along the vehicle path. The behavior selector 706 then determines a vehicle behavior for the vehicle based on the expected road elements. Example behaviors include slowing down, adjusting steering, or adjusting pressure in an active suspension system to prevent uncomfortable vertical acceleration that would otherwise result from driving over a pothole.

[0108] As Figure 7As shown, the behavior selector 706 provides the selected vehicle behavior to a motion planner 708 and / or a controller 710. In particular, if the selected vehicle behavior is a vehicle trajectory, the behavior selector 706 provides the vehicle behavior to the motion planner 708. The motion planner 708 plans vehicle motion based on the selected vehicle trajectory. The motion planner 708 then provides instructions to the controller 710. And if the selected vehicle behavior is a vehicle setting, the behavior selector 706 provides the vehicle behavior to the controller 710. In some examples, the behavior selector 706 provides the selected vehicle behavior to both the motion planner 708 and the controller 710.

[0109] Figure 8 A block diagram of an example system 800 for surface localization correction and sensor damage detection is illustrated. The system 800 is configured to determine whether a surface localization is inaccurate based on an identified surface element. Additionally, the system 800 is configured to determine whether a vehicle sensor is damaged based on the identified surface element. Moreover, the system 800 is configured to perform a corrective action in response to determining that the surface localization is inaccurate and / or that the vehicle sensor is damaged.

[0110] As Figure 8 illustrated, the system 800 includes a localization sensor suite 802, a localization module 804, a signal processor 806, a surface element matcher 808, and a fault response module 810. It should be noted that the system 800 is shown for illustrative purposes only, as the system 800 can include additional components and / or remove one or more components without departing from the scope of the present disclosure. Moreover, the components of the system 800 can be arranged and connected in any manner.

[0111] In embodiments, the localization sensor suite 802 includes at least one localization sensor that generates localization data. Example localization sensors include GPS, IMU, sensors that measure vehicle linear acceleration and angular rates, wheel rate sensors to measure or estimate wheel slip, wheel brake pressure or brake torque sensors, engine torque or wheel torque sensors, steering angle sensors, and other sensors such as LiDAR. The localization sensor suite 802 provides localization data to the localization module 804.

[0112] In an embodiment, the positioning module 804 determines the vehicle position by calculating the position using data from the positioning sensor suite 802. For example, the positioning module 804 uses data from a GNSS (Global Navigation Satellite System) sensor and geographic data to calculate the longitude and latitude of the vehicle. The geographic data is associated with one or more objects (e.g., buildings and / or trees, etc.) within a distance from the vehicle (e.g., the geographic data represents one or more objects within a distance from the vehicle). The geographic data is determined, for example, at least in part based on data from the LiDAR system of the vehicle. The data used by the positioning module 804 include a high-precision map of the geometric attributes of the lane, a map describing the road network connection attributes, and a map describing the physical attributes of the lane. In an embodiment, the positioning module 804 uses location data to generate a map of the spatial location of the surface elements identified by the vehicle or received from a shared database. The surface elements included in the map are called expected surface elements.

[0113] In an embodiment, the signal processor 806 is connected to Figure 7 The signal processor 806 is the same or similar to the signal processor 704. Like the signal processor 704, the signal processor 806 is configured to perform operations related to processing sensor data. In particular, the signal processor 806 receives data from the surface sensor suite ( Figure 8 The signal processor 806 detects the road elements based on the received sensor data.

[0114] In an embodiment, the surface element matcher 808 is configured to determine whether a vehicle sensor is damaged or whether the surface positioning is inaccurate based on the detected surface element. To do so, the surface element matcher 808 receives information indicating the detected surface element from the signal processor 806. This information includes the type of the surface element and the spatial location of the surface element. The surface element matcher 808 also receives expected surface elements in the area where the surface element was detected from the positioning module 804. The surface element matcher 808 determines whether the type of the detected surface element is similar to or identical to the type of one of the expected surface elements in the area. If the surface element matcher 808 determines that there is no match, the surface element matcher 808 determines whether the vehicle sensor has previously received sensor data from the area. If the vehicle sensor has previously received sensor data from the area and estimates that the surface element is present at this time (e.g., the surface element is not temporary), the surface element matcher 808 determines that the vehicle sensor is damaged. The surface element matcher 808 then provides a sensor fault detection signal to the fault response module 810.

[0115] In contrast, if the surface element matcher 808 determines that the type of the detected surface element matches the type of the expected surface element, the surface element matcher 808 compares the spatial location of the detected surface element to the spatial location of the expected surface element. If the spatial locations are similar or identical to each other, the surface element matcher 808 determines that the surface localization is accurate. However, if the spatial locations do not match, the surface element matcher 808 determines that the surface localization is inaccurate. In response, the surface element matcher 808 sends a localization correction signal to the localization module 804. The localization correction signal includes the detected spatial location of the surface element. Additionally or alternatively, the surface element matcher 808 sends a sensor calibration correction signal to the localization sensor suite 802. The localization sensor suite 802 uses the sensor calibration correction signal to calibrate the localization sensor that computed the spatial location of the expected surface element.

[0116] In embodiments, the surface element matcher 808 uses the comparison of the detected road element to the expected road element to reduce false positives of collision detection. In examples, if the surface element matcher 808 determines that the detected road element matches the expected road element, the surface element matcher 808 determines that the vehicle did not drive over an unknown object or that a collision did not occur.

[0117] In embodiments, the fault response module 810 is configured to perform a corrective action in response to receiving a sensor fault detection signal from the surface element matcher 808. The fault response module 810 is responsible for mapping the fault to a corrective action and triggering the corrective action. For example, if a collision is detected (or suspected) to have occurred, the vehicle can be commanded to stop on the side of the travel lane (to avoid a hit-and-run scenario or to wait for a remote operator intervention). In contrast, if the fault is a sensor fault, or if the sensor is redundant or non-critical in the context of a multi-modal sensor suite, a less urgent maintenance action is prescribed. For example, the vehicle is directed to no longer accept additional passenger requests and to return to a maintenance facility.

[0118] Figure 9 A flowchart illustrating a process 900 for surface guidance decision making is shown. The process 900 can be performed by the system 700 or the system 800. Sensor measurements indicative of at least one of sound and vibration associated with a road element are received (902) from at least one sensor of a vehicle. In examples, the at least one sensor of the vehicle includes the sensors 702 of the system 700. A road element is identified (904) based on a pattern in the sensor measurements. In examples, the pattern in the sensor measurements is extracted by the signal processor 704 of the system 700 or the signal processor 806 of the system 800. Further, the signal processor 704 or the signal processor 806 identifies the road element based on the extracted pattern. Figure 7 Figure 7 Figure 8 In addition, the signal processor 704 or the signal processor 806 identifies the road element based on the extracted pattern.​​

[0119] A vehicle behavior of the vehicle is determined (906) based on the road element. In an example, the vehicle behavior is determined by the behavior selector 706 of the vehicle 700. The vehicle behavior includes a vehicle trajectory and / or a vehicle driving setting, and is selected by the behavior selector 706 of the vehicle 700 based on the road element. Figure 7 Figure 7 The vehicle is controlled (908) to operate in accordance with the vehicle driving behavior. In an example, the vehicle is controlled by the controller 710 of the vehicle 700. Figure 7

[0120] In some implementations, the process 900 further includes adding a representation of the road element to a shared map. The representation includes at least one of: a spatial location of the road element, the sensor measurement, a pattern in the sensor measurement, and the vehicle behavior. In an example, the shared map can be updated and accessed by a fleet of vehicles.

[0121] In some implementations, the road element is associated with a location, and the process 900 further involves: comparing the measured road element to expected road elements in the location; determining that the road element does not match any of the expected road elements; and in response, determining a fault of a sensor that previously performed measurements at the location. In an example, these operations are performed by the surface element matcher 808 of the vehicle 700. Figure 8 In some examples, a consistent difference between the map and recent observations can instead indicate a change in infrastructure rather than a sensor fault (e.g., a new pothole due to wear on the road surface). In such examples, if the difference is observed from multiple vehicles, evidence of a map change can be determined.

[0122] In some implementations, the sensor is associated with the vehicle or another vehicle.

[0123] In some implementations, the road element is associated with a location, the sensor measurement is a first sensor measurement, and the process 900 involves: comparing the road element to expected road elements in the location; determining a threshold difference between the first sensor measurement and a second sensor measurement associated with an expected road element; and in response to determining the threshold difference, determining that a sensor that performed the second sensor measurement is not calibrated. In an example, these operations are performed by the surface element matcher 808 of the vehicle 700. Figure 8

[0124] ​​​In some implementations, the process 900 further involves: reobserving the location of the road element; comparing the recently observed road element to an expected road element in the location; determining that the recently observed road element matches the first expected road element; or, conversely, determining a difference between the location of the recently observed road element and the stored location of the first expected road element; and updating the stored location of the first expected road element based on the location of the recently observed road element.

[0125] In some implementations, the vehicle behavior is a first vehicle behavior, and the process 900 further involves: obtaining, from the shared database, an expected road element along a road path of the vehicle; and determining a second vehicle behavior based on the expected road element.

[0126] In some implementations, the at least one sensor includes at least one of: a vibration sensor, a tilt sensor, a shock sensor, an accelerometer, a gyroscope, a tire pressure sensor, a suspension pressure sensor, a strain sensor, a sound pressure sensor, and a microphone.

[0127] In some implementations, the process 900 further involves: comparing the road element to an expected road element in the location; and in response to identifying a match of the road element, determining that the vehicle did not drive over an unknown object or that a collision did not occur.

[0128] In some implementations, identifying the road element based on the pattern in the sensor measurements involves: comparing the sensor measurements to historical patterns stored in a database, where the historical patterns are associated with respective road elements; identifying a first historical pattern that has a threshold level of similarity to the pattern; and determining that the respective road element associated with the first historical pattern is the road element.

[0129] In the foregoing description, embodiments of the application have been described with reference to a number of specific details that can vary depending on implementation. Thus, the specification and drawings should be construed in an illustrative sense rather than a restrictive sense. The sole and exclusive indicator of the scope of the application, and what is intended by the applicants to be the scope of the application, is the literal and equivalent scope of the claims as issued by the US Patent and Trademark Office, including any subsequent correction. Any definitions of terms here in this detailed description are expressly incorporated by reference from the corresponding portions of the claims. In addition, when the description or claims use the singular "a", "an" or "the" it should be taken as meaning "at least one" and not as limiting to a single element. Furthermore, the words "comprise", "comprising", "comprises", "include", "including", "includes", "contain", "containing", "contains", "have", "having", "has", "may", "might", "must", "can", "could", "should", "will", "would", "but", "and", "or", "by", "against", "into", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto", "onto",

Claims

1. A system for a vehicle, comprising: at least one sensor of the vehicle; at least one computer-readable medium storing computer-executable instructions; at least one processor configured to execute the computer-executable instructions, the execution to operate, the operations comprising: receiving, from the at least one sensor, sensor measurements indicative of at least one of sound and vibrations associated with a road element, wherein the road element is a raised surface or a surface depression on a road surface; identifying the road element based on a pattern in the sensor measurements, wherein the pattern comprises an amplitude and a direction expectation of a signal in the sensor measurements; determining a location of the road element; comparing the road element to expected road elements in the location; determining that the road element matches a first expected road element; determining a difference between the location of the road element and a stored location of the first expected road element; updating the stored location of the first expected road element based on the location of the road element; determining, based on the road element, that the vehicle is encountering a particular scenario; determining a vehicle behavior based on the particular scenario; and controlling the vehicle to operate in accordance with the vehicle behavior.

2. The system of claim 1, the operations further comprising: adding a representation of the road element to a shared map, wherein the representation of the road element comprises at least one of: a spatial location of the road element, the sensor measurements, the pattern in the sensor measurements, and the vehicle behavior.

3. The system of claim 1, wherein, the road element is associated with a location, and wherein the operations further comprise: comparing the road element to expected road elements in the location; determining that the road element does not match any of the expected road elements; and in response, determining a malfunction of a sensor that previously measured in the location.

4. The system of claim 3, wherein, the sensor is associated with the vehicle or other vehicles.

5. The system of claim 1, wherein, the road element is associated with a location, wherein the sensor measurements are first sensor measurements, and wherein the operations further comprise: comparing the road element to expected road elements in the location; determining a threshold difference between the first sensor measurements and second sensor measurements associated with the expected road elements; and in response to determining the threshold difference, determining that a sensor that took the second sensor measurements is not calibrated.

6. The system of claim 5, wherein, the sensor is associated with the vehicle or other vehicles.

7. The system of claim 1, wherein, the vehicle behavior is a first vehicle behavior, and wherein the operations further comprise: obtaining, from a shared database, expected road elements along a road path of the vehicle; and determining a second vehicle behavior based on the expected road elements.

8. The system of claim 1, wherein, the at least one sensor comprises at least one of: a vibration sensor, a tilt sensor, a shock sensor, an accelerometer, a gyroscope, a tire pressure sensor, a suspension pressure sensor, a spread sensor, a sound pressure sensor, and a microphone.

9. The system of claim 1 , wherein the operations comprise: comparing the road element to expected road elements in the location; as well as In response to identifying a match to the road element, it is determined that the vehicle has not passed over an unknown object or has not collided.

10. The system of claim 1, wherein, Identifying the road element based on patterns in the sensor measurements includes: comparing the sensor measurements with historical patterns stored in a database, wherein the historical patterns are associated with corresponding road elements; identifying a first historical pattern having a threshold level of similarity to the pattern; and The corresponding road element determined to be associated with the first history pattern is the road element.

11. A method for a vehicle, comprising: receiving sensor measurements from at least one sensor of the vehicle indicative of at least one of sound and vibration associated with a road element, wherein the road element is a raised surface or a surface depression on a road surface; identifying the road element based on a pattern in the sensor measurements, wherein the pattern includes an expectation of magnitude and direction of a signal in the sensor measurements; determining a location of the road element; comparing the road element to expected road elements in the location; Determining that the road element matches a first expected road element; determining a difference between the location of the road element and a stored location of the first expected road element; and updating the stored location of the first expected road element based on the location of the road element; Determining based on the road elements that the vehicle is encountering a specific scenario; determining vehicle behavior based on the specific scenario; and The vehicle is controlled to operate according to the vehicle behavior.

12. A non-transitory computer-readable storage medium comprising at least one program, the at least one program being executed by at least one processor of a first device, the at least one program comprising instructions that, when executed by the at least one processor, cause the first device to perform the method according to claim 11.

13. A computer program product comprising at least one program, the at least one program being executed by at least one processor of a first device, the at least one program comprising instructions which, when executed by the at least one processor, cause the first device to perform the method according to claim 11.

Images