Method and storage medium for a vehicle

Abstract

The invention relates to methods and storage media for a vehicle. Techniques for spatially and temporally consistent ground modeling with information fusion are described. A vehicle location and orientation are obtained, and a transient ground height estimate for an anchor cell of a grid map for the vehicle is obtained based on the obtained vehicle location and orientation. A pseudo ground height associated with a non-anchor cell of the grid map is computed. A Bayesian filtering framework is used to generate a final ground height estimate for the anchor cell and the non-anchor cell based on the transient ground height estimate for the anchor cell, the pseudo ground height associated with the non-anchor cell, an estimated ground height from a previous timestamp, or any combination thereof. The vehicle operates in accordance with the final ground height estimate.

Description

Technical Field

[0001] This description relates to the use of information fusion for spatially and temporally consistent ground modeling. Background Technology

[0002] Ground level estimation is used to estimate the ground level at a consistent location in an environment. Ground level estimation describes the characteristics associated with the surface on which a vehicle travels. In some cases, ground level estimation corresponds to a specific slope of the surface. Summary of the Invention

[0003] A method for a vehicle includes: obtaining the vehicle's location and orientation in an environment using a positioning system; obtaining an instantaneous ground height estimate of anchor cells of a raster map for the vehicle based on the obtained vehicle location and orientation using at least one processor; calculating pseudo ground heights associated with non-anchor cells of the raster map using the at least one processor; generating a final ground height estimate for the anchor cells and the non-anchor cells using a Bayesian filtering architecture based on the instantaneous ground height estimate for the anchor cells, the pseudo ground heights associated with the non-anchor cells, an estimated ground height from a previous timestamp, or any combination thereof; and operating the vehicle using a controller based on the final ground height estimate.

[0004] A method for a vehicle includes: receiving raster map data including anchor units and non-anchor units using at least one processor of the vehicle; receiving ground elevations of the anchor units and the non-anchor units using the at least one processor; for the non-anchor units: determining the nearest neighbor anchor unit for the non-anchor unit using the at least one processor; determining a single pseudo-ground elevation for the non-anchor unit using the at least one processor based on pseudo-ground elevations calculated using at least the ground elevations of each respective nearest neighbor anchor unit, wherein the single pseudo-ground elevation is associated with uncertainty; calculating at least one weighting factor using the at least one processor based on the associated uncertainty; combining the pseudo-ground elevations according to the at least one weighting factor using the at least one processor to generate a final pseudo-ground elevation for the non-anchor unit; iteratively filtering an instantaneous ground elevation estimate and the final pseudo-ground elevation using the at least one processor to obtain a final ground elevation estimate; and navigating the vehicle using control circuitry of the vehicle based on the final ground elevation estimate.

[0005] A non-transitory computer-readable storage medium storing instructions that, when executed by at least one programmable computer processor of a vehicle, cause the at least one programmable computer processor to perform the following operations: obtaining the location and orientation of the vehicle in an environment; obtaining an instantaneous ground height estimate of anchor cells of a grid map for the vehicle based on the obtained vehicle location and orientation; calculating pseudo ground heights associated with non-anchor cells of the grid map; generating a final ground height estimate for the anchor cells and the non-anchor cells based on the instantaneous ground height estimate for the anchor cells, the pseudo ground heights associated with the non-anchor cells, an estimated ground height from a previous timestamp, or any combination thereof; and operating the vehicle according to the final ground height estimate. Attached Figure Description

[0006] Figure 1 An example of an autonomous vehicle (AV) with autonomous capabilities is shown.

[0007] Figure 2 An example "cloud" computing environment is shown.

[0008] Figure 3 The computer system is shown.

[0009] Figure 4 An example architecture for AV is shown.

[0010] Figure 5 Examples of inputs and outputs that a sensing system can use are shown.

[0011] Figure 6 An example of a LIDAR system is shown.

[0012] Figure 7 The image shows a LIDAR system in operation.

[0013] Figure 8 Additional details on the operation of the LIDAR system are shown.

[0014] Figure 9 A block diagram illustrating the relationship between the inputs and outputs of the planning system.

[0015] Figure 10 This shows the directed graph used in path planning.

[0016] Figure 11 A block diagram showing the inputs and outputs of the control system is provided.

[0017] Figure 12 A block diagram showing the controller's inputs, outputs, and components is provided.

[0018] Figure 13A It is an aerial view of a scene with obscured areas.

[0019] Figure 13B This is a diagram illustrating an example ground detection algorithm that operates on azimuth bins.

[0020] Figure 14 It is a block diagram of a system that uses information fusion to perform spatially and temporally consistent ground modeling.

[0021] Figure 15A It is an illustration of a two-dimensional ground elevation grid map.

[0022] Figure 15B It is an illustration generated by pseudo-ground height measurement of a raster map.

[0023] Figure 16A It is a flowchart of the process of using information fusion to perform spatially and temporally consistent ground modeling.

[0024] Figure 16B It is a flowchart of the process of creating a spatially and temporally consistent ground model by fusing information from multiple timestamps.

[0025] Figure 17 This is a flowchart of the process for implementing pseudo-ground height measurement. Detailed Implementation

[0026] In the following description, numerous specific details are set forth for purposes of explanation in order to provide a thorough understanding of this disclosure. However, it will be apparent that this disclosure may be practiced without these specific details. In other instances, well-known constructions and apparatuses are shown in block diagram form to avoid unnecessarily obscuring this disclosure.

[0027] In the accompanying drawings, for ease of description, a specific arrangement or order of schematic elements (such as those representing devices, modules, systems, instruction blocks, and data elements) is shown. However, those skilled in the art will understand that the specific order or arrangement of the schematic elements in the drawings is not intended to imply a requirement for a particular processing order or sequence, or a separation of processing procedures. Furthermore, the inclusion of schematic elements in the drawings is not intended to imply that such elements are required in all embodiments, nor is it intended to imply that features represented by such elements cannot be included in some embodiments or cannot be combined with other elements in some embodiments.

[0028] Furthermore, in the accompanying drawings, connecting elements, such as solid or dashed lines or arrows, are used to illustrate connections, relationships, or associations between two or more other schematic elements. The absence of any such connecting element does not imply that connections, relationships, or associations cannot exist. In other words, connections, relationships, or associations between some elements are not shown in the drawings so as not to obscure the content of this disclosure. Additionally, for ease of illustration, a single connecting element is used to represent multiple connections, relationships, or associations between elements. For example, if a connecting element represents communication of signals, data, or instructions, those skilled in the art will understand that such an element represents one or more signal paths (e.g., a bus) that may be necessary to affect the communication.

[0029] Reference will now be made in detail to embodiments, examples of which are illustrated in the accompanying drawings. Numerous specific details are set forth in the following detailed description in order to provide a thorough understanding of the various embodiments described. However, it will be apparent to those skilled in the art that the various embodiments described can be practiced without these specific details. In other instances, well-known methods, procedures, components, circuits, and networks have not been described in detail so as not to unnecessarily obscure aspects of the embodiments.

[0030] The features described below can each be used independently of each other or in any combination with other features. However, any individual feature may not solve any of the problems discussed above, or may only solve one of the problems discussed above. Some of the problems discussed above may not be adequately solved by any of the features described herein. Although headings are provided, information relating to specific headings but not found in the sections bearing those headings can be found elsewhere in this specification. Embodiments are described herein based on the following summary:

[0031] 1. General Overview

[0032] 2. System Overview

[0033] 3. AV Architecture

[0034] 4. AV Input

[0035] 5. AV Planning

[0036] 6. AV Control

[0037] 7. Ground Modeling

[0038] 8. A system for spatially and temporally consistent ground modeling using information fusion.

[0039] 9. Utilizing information fusion to perform spatially and temporally consistent ground modeling.

[0040] General Overview

[0041] Mobile robots, such as autonomous vehicles, rely on one or more sensors to perceive their surroundings. Environmental perception includes, but is not limited to, modeling the ground surface. Ground height typically refers to the height of the ground relative to the vehicle's current attitude, where the ground is the solid surface on which the vehicle can traverse. In some cases, ground height perception by the vehicle's sensors may be unavailable due to occlusion or areas exceeding the range of one or more sensors. Pseudo-ground height measurements are generated to produce spatially consistent ground height estimates. Several techniques for obtaining additional ground height measurements can be used to generate ground models in the presence of occlusion or out-of-range areas. Pseudo-measurements are fused with other ground height measurements to obtain a single, spatially and temporally consistent ground height model.

[0042] Some advantages of these techniques include obtaining spatially and temporally consistent ground height estimates. In particular, temporal consistency is achieved through the propagation of ground height over time. Furthermore, the pseudo-ground height measurements according to this technique incorporate information from a nearby reliable context, resulting in a consistent ground height model throughout the modeling space. Moreover, this technique can fuse ground height information from various sources, thereby producing a robust ground model that takes into account various occlusions and hardware limitations.

[0043] System Overview

[0044] Figure 1 An example of an autonomous AV 100 with autonomous capabilities is shown.

[0045] As used herein, the term “autonomy” refers to a function, feature, or facility that enables a vehicle to operate partially or fully without real-time human intervention, including but not limited to full AV, high AV, and conditional AV.

[0046] As used in this article, an autonomous vehicle (AV) is a vehicle with autonomous capabilities.

[0047] As used in this article, "vehicle" includes any mode of transport for goods or people. Examples include cars, buses, trains, airplanes, drones, trucks, ships, vessels, submersibles, and spacecraft. Driverless cars are an example of vehicles.

[0048] As used herein, a “track” refers to a path or route that navigates an AV from a first spatiotemporal location to a second spatiotemporal location. In embodiments, the first spatiotemporal location is referred to as the initial location or starting point, and the second spatiotemporal location is referred to as the destination, final location, target, target location, or target position. In some examples, a track consists of one or more road segments (e.g., segments of a road), and each road segment consists of one or more blocks (e.g., a lane or part of an intersection). In embodiments, spatiotemporal locations correspond to real-world locations. For example, a spatiotemporal location is a pick-up or drop-off point for people or goods to board or alight.

[0049] As used herein, “(one or more) sensor” includes one or more hardware components for detecting information relating to the environment surrounding the sensor. Some hardware components may include sensing components (e.g., image sensors, biometric sensors), transmission and / or receiving components (e.g., laser or radio frequency wave transmitters and receivers), electronic components (such as analog-to-digital converters), data storage devices (such as RAM and / or non-volatile memory), software or firmware components, and data processing components (such as application-specific integrated circuits), microprocessors, and / or microcontrollers.

[0050] As used herein, a “scene description” is a data structure (e.g., a list) or data stream that includes one or more classified or labeled objects detected by one or more sensors on an AV vehicle, or one or more classified or labeled objects provided by a source outside the AV.

[0051] As used in this article, a "road" is a physical area that can be traversed by vehicles and can correspond to a named passageway (e.g., a city street, an interstate highway, etc.) or an unnamed passageway (e.g., a driveway within a house or office building, a section of a parking lot, a section of an vacant parking lot, a waste disposal route in a rural area, etc.). Because some vehicles (e.g., four-wheel drive pickup trucks, SUVs, etc.) can traverse a variety of physical areas that are not particularly suitable for vehicle travel, a "road" can be any physical area that is not formally defined as a passageway by any municipality or other government or administrative agency.

[0052] As used herein, a “lane” is the portion of a road through which vehicles can pass. Sometimes lanes are identified based on lane markings. For example, a lane may correspond to most or all of the space between lane markings, or only a portion of the space between lane markings (e.g., less than 50%). For instance, a road with widely spaced lane markings may accommodate two or more vehicles, allowing one vehicle to overtake another without crossing the lane markings; therefore, this could be interpreted as a lane being narrower than the space between lane markings, or as having two lanes. Lanes can also be interpreted in the absence of lane markings. For example, a lane may be defined based on the physical characteristics of the environment (e.g., rocks and trees along a main road in a rural area, or natural obstacles that should be avoided, for example, in underdeveloped areas). Lanes can also be interpreted independently of lane markings or physical characteristics. For example, a lane may be interpreted based on any unobstructed path in an area that would otherwise lack the features that would be interpreted as lane boundaries. In the example scenario, an AV could interpret a lane as passing through an unobstructed portion of a field or open space. In another example scenario, an AV can interpret lanes that pass through a wide road (e.g., wide enough for two or more lanes) without lane markings. In this scenario, the AV can communicate lane-related information to other AVs, allowing them to coordinate route planning using the same lane information.

[0053] The term “over-the-air (OTA) client” includes any AV, or any electronic device embedded in, coupled to, or communicating with an AV (e.g., computer, controller, IoT device, electronic control unit (ECU)).

[0054] The term "over-the-air (OTA) update" means any update, alteration, deletion, or addition to software, firmware, data, or configuration settings, or any combination thereof, delivered to an OTA client using proprietary and / or standardized wireless communication technologies, including but not limited to: cellular mobile communications (e.g., 2G, 3G, 4G, 5G), radio local area networks (e.g., WiFi), and / or satellite Internet.

[0055] The term "edge node" refers to one or more edge devices coupled to a network that provide a portal for communicating with AV and can communicate with other edge nodes and cloud-based computing platforms to schedule OTA updates and deliver OTA updates to OTA clients.

[0056] The term "edge device" refers to a device that implements an edge node and provides a physical wireless access point (AP) to the core network of an enterprise or service provider (such as Verizon or AT&T). Examples of edge devices include, but are not limited to: computers, controllers, transmitters, routers, routing switches, integrated access devices (IADs), multiplexers, metropolitan area network (MAN) and wide area network (WAN) access devices.

[0057] "One or more" includes functions performed by one element, functions performed by multiple elements, such as in a distributed manner, several functions performed by one element, several functions performed by several elements, or any combination of the above.

[0058] It will also be understood that, although in some cases the terms “first,” “second,” etc., are used herein to describe various elements, these elements should not be limited by these terms. These terms are used only to distinguish one element from another. For example, without departing from the scope of the various described embodiments, a first contact may be referred to as a second contact, and similarly, a second contact may be referred to as a first contact. Both the first contact and the second contact are contacts, but they are not the same contact.

[0059] The terminology used in the description of the various embodiments described herein is for the purpose of describing particular embodiments only and is not intended to be limiting. As used in the description of the various embodiments described and the appended claims, the singular forms “a,” “an,” and “the” are also intended to include the plural forms, unless the context clearly indicates otherwise. It will also be understood that “and / or,” as used herein, refers to and includes any and all possible combinations of one or more related list items. It will also be understood that when the terms “comprising,” “including,” “possessing,” and / or “having” are used in this specification, they specifically indicate the presence of the stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0060] As used herein, depending on the context, the term "if" may optionally be understood as meaning "when" or "at that time" or "in response to being determined" or "in response to being detected." Similarly, depending on the context, the phrase "if determined" or "if [the stated condition or event] has been detected" may optionally be understood as meaning "when determined" or "in response to being determined" or "when [the stated condition or event] is detected" or "in response to being detected."

[0061] As used herein, an AV system refers to an AV and an array of hardware, software, stored data, and real-time generated data that support AV operation. In embodiments, the AV system is incorporated within an AV. In embodiments, the AV system is distributed across several locations. For example, some of the software of the AV system is similar to that described below. Figure 2 The cloud computing environment described is implemented in the cloud computing environment of 200.

[0062] Generally, this document describes technologies applicable to any vehicle with one or more autonomous capabilities, including fully automated driving (AV), highly automated driving (AV), and conditionally automated driving (AV), such as so-called Level 5, Level 4, and Level 3 vehicles, respectively (see SAE International Standard J3016: Classification and Definition of Terms Related to Automated Driving Systems for Motor Vehicles on Roads, the entire contents of which are incorporated herein by reference for further details on vehicle autonomy levels). The technologies described in this document are also applicable to partially automated driving (AV) and driver-assisted vehicles, such as so-called Level 2 and Level 1 vehicles (see SAE International Standard J3016: Classification and Definition of Terms Related to Automated Driving Systems for Motor Vehicles on Roads). In embodiments, one or more Level 1, Level 2, Level 3, Level 4, and Level 5 vehicle systems can automatically perform certain vehicle operations (e.g., steering, braking, and map usage) under certain operating conditions based on the processing of sensor inputs. The technologies described in this document can benefit vehicles of any level, ranging from fully automated driving to human-operated vehicles.

[0063] AVs have advantages over vehicles that require human drivers. One advantage is safety. For example, in 2016, the U.S. experienced 6 million car accidents, 2.4 million injuries, 40,000 deaths, and 13 million vehicle collisions, with an estimated social cost of over $910 billion. From 1965 to 2015, the number of traffic fatalities per 100 million miles driven in the U.S. decreased from about 6 to about 1, partly due to additional safety features deployed in vehicles. For example, an extra half-second of warning associated with an impending collision is believed to mitigate 60% of front and rear collisions. However, passive safety features (such as seat belts and airbags) may have reached their limits in improving these figures. Therefore, active safety measures, such as automated vehicle controls, are a possible next step in improving these statistics. Since human drivers are considered to be responsible for serious pre-collision events in 95% of collisions, autonomous driving systems could potentially achieve better safety outcomes by: identifying and avoiding emergencies more reliably than humans; making better decisions, obeying traffic regulations better than humans, and predicting future events better than humans; and controlling vehicles more reliably than humans.

[0064] refer to Figure 1 The AV system 120 enables the vehicle 100 to operate along a trajectory 198, traversing the environment 190 to the destination 199 (sometimes referred to as the final location), while avoiding objects (e.g., natural obstacles 191, vehicles 193, pedestrians 192, cyclists and other obstacles) and complying with road rules (e.g., operating rules or driving preferences).

[0065] In an embodiment, the AV system 120 includes means 101 for receiving and operating operation commands from and on a computer processor 146. The term "operation command" is used to refer to executable instructions (or a set of instructions) that cause a vehicle to perform actions (e.g., driving maneuvers). Operation commands may, without limitation, include instructions for causing the vehicle to begin moving forward, stop moving forward, begin moving backward, stop moving backward, accelerate, decelerate, make a left turn, and make a right turn. In an embodiment, the computer processor 146 is referenced below. Figure 2 The processor 204 described is similar. Examples of the device 101 include a steering controller 102, a brake 103, a gear, an accelerator pedal or other acceleration control mechanism, a windshield wiper, a side door lock, a window controller, and a turn indicator.

[0066] In an embodiment, the AV system 120 includes sensors 121 for measuring or inferring attributes of the state or condition of the vehicle 100, such as the AV's position, linear velocity and angular velocity, linear acceleration and angular acceleration, and heading (e.g., the direction of the front end of the vehicle 100). Examples of sensors 121 are GPS, inertial measurement units (IMUs) that measure both linear acceleration and angular rate of the vehicle, wheel rate sensors for measuring or estimating wheel slip ratio, wheel braking pressure or braking torque sensors, engine torque or wheel torque sensors, and steering angle and angular rate sensors.

[0067] In an embodiment, sensor 121 also includes sensors for sensing or measuring properties of the AV's environment. Examples include a monocular or stereo camera 122 with visible, infrared, or thermal (or both) spectra, a LiDAR 123, a RADAR, an ultrasonic sensor, a time-of-flight (TOF) depth sensor, a rate sensor, a temperature sensor, a humidity sensor, and a precipitation sensor.

[0068] In one embodiment, the AV system 120 includes a data storage unit 142 and a memory 144 for storing machine instructions associated with a computer processor 146 or data collected by the sensor 121. In another embodiment, the data storage unit 142 is associated with the following... Figure 3The described ROM 308 or storage device 310 is similar. In this embodiment, memory 144 is similar to main memory 306 described below. In this embodiment, data storage unit 142 and memory 144 store historical, real-time, and / or predictive information about environment 190. In this embodiment, the stored information includes maps, driving performance, traffic congestion updates, or weather conditions. In this embodiment, data related to environment 190 is transmitted from remote database 134 to vehicle 100 via a communication channel.

[0069] In an embodiment, the AV system 120 includes communication devices 140 for transmitting measured or inferred attributes of the state and conditions of other vehicles, such as position, linear velocity and angular velocity, linear acceleration and angular acceleration, and linear heading and angular heading, to vehicle 100. These devices include vehicle-to-vehicle (V2V) and vehicle-to-infrastructure (V2I) communication devices, as well as devices for wireless communication via point-to-point or ad hoc networks, or both. In an embodiment, communication device 140 communicates across the electromagnetic spectrum (including radio and optical communications) or other media (e.g., air and acoustic media). The combination of vehicle-to-vehicle (V2V), vehicle-to-infrastructure (V2I) communication (and in some embodiments, one or more other types of communication) is sometimes referred to as vehicle-to-all-things (V2X) communication. V2X communication typically conforms to one or more communication standards for communication with and between AVs.

[0070] In an embodiment, the communication device 140 includes a communication interface. For example, this may be a wired, wireless, WiMAX, Wi-Fi, Bluetooth, satellite, cellular, optical, near-field, infrared, or radio interface. The communication interface transmits data from a remote database 134 to the AV system 120. In an embodiment, the remote database 134 is embedded in, for example... Figure 2 In the aforementioned cloud computing environment, communication device 140 transmits data collected from sensor 121 or other data related to the operation of vehicle 100 to remote database 134. In some embodiments, communication device 140 transmits information related to teleoperation to vehicle 100. In some embodiments, vehicle 100 communicates with other remote (e.g., "cloud") servers 136.

[0071] In this embodiment, the remote database 134 also stores and transmits digital data (e.g., data such as road and street locations). This data is stored in memory 144 on the vehicle 100 or transmitted from the remote database 134 to the vehicle 100 via a communication channel.

[0072] In one embodiment, the remote database 134 stores and transmits historical information (e.g., rate and acceleration distribution) related to driving attributes of vehicles that previously traveled along trajectory 198 at similar times of day. In one implementation, such data may be stored in memory 144 on vehicle 100 or transmitted from the remote database 134 to vehicle 100 via a communication channel.

[0073] The computer processor 146 located on the vehicle 100 generates control actions in an algorithmic manner based on both real-time sensor data and prior information, allowing the AV system 120 to perform its autonomous driving capabilities.

[0074] In one embodiment, the AV system 120 includes a computer peripheral device 132 coupled to a computer processor 146 for providing information and alerts to a user of the vehicle 100 (e.g., a passenger or a remote user) and receiving input from that user. In another embodiment, the peripheral device 132 is similar to the one described in the following reference. Figure 3 The discussed display 312, input device 314, and cursor controller 316 are coupled wirelessly or wiredly. Any two or more interface devices can be integrated into a single device.

[0075] In one embodiment, the AV system 120 receives and enforces a privacy level for an occupant, such as one specified by the occupant or stored in a profile associated with the occupant. The occupant's privacy level determines how access is permitted to specific occupant-related information (e.g., occupant comfort data, biometric data, etc.) stored in the occupant profile and / or stored on cloud server 136 and associated with the occupant profile. In one embodiment, the privacy level specifies specific occupant-related information that is deleted once the ride is complete. In another embodiment, the privacy level specifies specific occupant-related information and identifies one or more entities authorized to access that information. Examples of the specified entities authorized to access the information may include other AVs, third-party AV systems, or any entity that could potentially access the information.

[0076] An occupant's privacy level can be specified at one or more granular levels. In an embodiment, the privacy level identifies specific information to be stored or shared. In an embodiment, the privacy level applies to all information associated with the occupant, allowing the occupant to specify that her personal information is not stored or shared. The designation of entities authorized to access specific information can also be specified at various granular levels. The various sets of entities authorized to access specific information may, for example, include other AVs, cloud server 136, specific third-party AV systems, etc.

[0077] In an embodiment, AV system 120 or cloud server 136 determines whether AV 100 or another entity can access certain information associated with an occupant. For example, a third-party AV system attempting to access occupant input related to a specific time and place must, for example, obtain authorization from AV system 120 or cloud server 136 to access information associated with the occupant. For example, AV system 120 uses a specified privacy level for the occupant to determine whether location- and time-related occupant input can be presented to a third-party AV system, AV 100, or another AV. This allows the occupant's privacy level to specify which other entities are allowed to receive data related to the occupant's actions or other data associated with the occupant.

[0078] Figure 2 This illustrates an example "cloud" computing environment. Cloud computing is a service delivery model that enables convenient, on-demand access over a network to a shared pool of configurable computing resources, such as networks, network bandwidth, servers, processing power, memory, storage, applications, virtual machines, and services. In a typical cloud computing system, one or more large cloud data centers house the machines used to deliver the services provided by the cloud. Now refer to... Figure 2 The cloud computing environment 200 includes cloud data centers 204a, 204b, and 204c interconnected via cloud 202. Data centers 204a, 204b, and 204c provide cloud computing services to computer systems 206a, 206b, 206c, 206d, 206e, and 206f connected to cloud 202.

[0079] A cloud computing environment 200 includes one or more cloud data centers. Generally speaking, a cloud data center (e.g.) Figure 2 The cloud data center 204a shown refers to the cloud (e.g., Figure 2 The physical arrangement of servers in cloud 202 (or a specific portion of the cloud) is illustrated. For example, servers are physically arranged in rooms, groups, rows, and racks within a cloud data center. A cloud data center has one or more regions, which include one or more server rooms. Each room has one or more rows of servers, and each row includes one or more racks. Each rack includes one or more individual server nodes. In some implementations, servers in regions, rooms, racks, and / or rows are arranged into groups based on the physical infrastructure requirements of the data center facility, including power, energy, heat, heat sources, and / or other requirements. In this embodiment, server nodes are similar to... Figure 3 The computer system described herein. Data center 204a has many computing systems distributed across multiple racks.

[0080] Cloud 202 includes cloud data centers 204a, 204b, and 204c, and networks and network resources (e.g., network devices, nodes, routers, switches, and network cables) for connecting cloud data centers 204a, 204b, and 204c and facilitating access to cloud computing services by computing systems 206a-f. In embodiments, the network represents one or more local networks, wide area networks, or any combination of wired or wireless networks coupled using terrestrial or satellite connections. Data exchanged over the network is transmitted using various network layer protocols, such as Internet Protocol (IP), Multiprotocol Label Switching (MPLS), Asynchronous Transfer Mode (ATM), Frame Relay, etc. Furthermore, in embodiments where the network represents a combination of multiple subnetworks, different network layer protocols are used on each underlying subnetwork. In some embodiments, the network represents one or more interconnected internetworks (such as the public Internet).

[0081] Computing systems 206a-f or cloud computing service consumers connect to the cloud 202 via network links and network adapters. In embodiments, computing systems 206a-f are implemented as various computing devices, such as servers, desktops, laptops, tablets, smartphones, Internet of Things (IoT) devices, AVs (including cars, drones, space shuttles, trains, buses, etc.), and consumer electronics. In embodiments, computing systems 206a-f are implemented in other systems or as part of other systems.

[0082] Figure 3 A computer system 300 is illustrated. In an implementation, the computer system 300 is a dedicated computing device. The dedicated computing device is hardwired to perform these technologies, or includes a digital electronic device such as one or more application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs) persistently programmed to perform the aforementioned technologies, or may include one or more general-purpose hardware processors programmed to perform these technologies according to program instructions in firmware, memory, other memory, or a combination thereof. Such a dedicated computing device may also combine custom hardwired logic, ASICs, or FPGAs with custom programming to accomplish these technologies. In various embodiments, the dedicated computing device is a desktop computer system, a portable computer system, a handheld device, a network device, or any other device that includes hardwired and / or program logic to implement these technologies.

[0083] In an embodiment, computer system 300 includes a bus 302 or other communication mechanism for conveying information, and a processor 304 coupled to the bus 302 to process information. Processor 304 is, for example, a general-purpose microprocessor. Computer system 300 also includes main memory 306, such as random access memory (RAM) or other dynamic storage device, coupled to the bus 302 to store information and instructions executed by processor 304. In one implementation, main memory 306 is used to store temporary variables or other intermediate information during the execution of instructions to be executed by processor 304. When these instructions are stored in a non-transitory storage medium accessible to processor 304, computer system 300 becomes a dedicated machine customized to perform the operations specified in the instructions.

[0084] In an embodiment, the computer system 300 further includes a read-only memory (ROM) 308 or other static storage device coupled to the bus 302 for storing static information and instructions of the processor 304. A storage device 310, such as a disk, optical disk, solid-state drive, or three-dimensional cross-point memory, is provided and coupled to the bus 302 to store information and instructions.

[0085] In this embodiment, the computer system 300 is coupled via a bus 302 to a display 312, such as a cathode ray tube (CRT), liquid crystal display (LCD), plasma display, light-emitting diode (LED) display, or an organic light-emitting diode (OLED) display for displaying information to a computer user. An input device 314, including alphanumeric keys and other keys, is coupled to the bus 302 for transmitting information and command selections to the processor 304. Another type of user input device is a cursor controller 316, such as a mouse, trackball, touchscreen, or cursor arrow keys, for transmitting directional information and command selections to the processor 304 and for controlling the movement of the cursor on the display 312. Such an input device typically has two degrees of freedom on two axes (a first axis (e.g., the x-axis) and a second axis (e.g., the y-axis)), which allow the device to specify a position in a plane.

[0086] According to one embodiment, the techniques described herein are executed by computer system 300 in response to processor 304 executing one or more sequences of one or more instructions contained in main memory 306. These instructions are read into main memory 306 from another storage medium, such as storage device 310. Executing the sequence of instructions contained in main memory 306 causes processor 304 to perform the process steps described herein. In alternative embodiments, hardwired circuitry is used instead of or in combination with software instructions.

[0087] As used herein, the term "storage medium" refers to any non-transitory medium that stores data and / or instructions that enable a machine to operate in a particular manner. Such storage media include non-volatile media and / or volatile media. Non-volatile media include, for example, optical discs, magnetic disks, solid-state drives, or three-dimensional cross-point memory such as storage device 310. Volatile media include dynamic memory, such as main memory 306. Common forms of storage media include, for example, floppy disks, floppy disks, hard disks, solid-state drives, magnetic tape or any other magnetic data storage medium, CD-ROMs, any other optical data storage medium, any physical medium with a perforated surface, RAM, PROMs and EPROMs, FLASH-EPROMs, NV-RAMs, or any other memory chips or memory cartridges.

[0088] Storage media differ from transmission media, but can be used in conjunction with them. Transmission media participate in the information transmission between storage media. For example, transmission media include coaxial cables, copper wires, and optical fibers, which include wires with a bus 302. Transmission media can also take the form of sound waves or light waves, such as those generated during radio wave and infrared data communication.

[0089] In embodiments, various forms of media involve carrying one or more sequences of one or more instructions to processor 304 for execution. For example, these instructions may initially be executed on a disk or solid-state drive of a remote computer. The remote computer loads the instructions into its dynamic memory and transmits them over a telephone line using a modem. A local modem of computer system 300 receives data over the telephone line and converts the data into an infrared signal using an infrared transmitter. An infrared detector receives the data carried in the infrared signal, and appropriate circuitry places the data on bus 302. Bus 302 carries the data to main memory 306, from which processor 304 retrieves and executes the instructions. The instructions received by main memory 306 may optionally be stored on storage device 310 before or after execution by processor 304.

[0090] Computer system 300 also includes a communication interface 318 coupled to bus 302. Communication interface 318 provides bidirectional data communication coupled to network link 320 connected to local network 322. For example, communication interface 318 is an Integrated Services Digital Network (ISDN) card, a cable modem, a satellite modem, or a modem used to provide data communication connectivity with a corresponding type of telephone line. As another example, communication interface 318 is a Local Area Network (LAN) card used to provide data communication connectivity with a compatible LAN. In some implementations, a wireless link is also implemented. In any such implementation, communication interface 318 transmits and receives electrical, electromagnetic, or optical signals carrying digital data streams representing various types of information.

[0091] Network link 320 typically provides data communication to other data devices via one or more networks. For example, network link 320 provides connectivity to host computer 324 or to a cloud data center or device operated by Internet Service Provider (ISP) 326 via local network 322. ISP 326, in turn, provides data communication services via a worldwide packet data communication network now commonly referred to as the "Internet" 328. Both local network 322 and Internet 328 use electrical, electromagnetic, or optical signals that carry digital data streams. Signals through various networks and signals on network link 320 via communication interface 318 are example forms of transmission media carrying digital data entering and leaving computer system 300. In embodiments, network 320 includes the aforementioned cloud 202 or a portion of cloud 202.

[0092] Computer system 300 sends messages and receives data including program code via (one or more) networks, network links 320, and communication interfaces 318. In an embodiment, computer system 300 receives code for processing. The received code is executed by processor 304 upon receipt and / or stored in storage device 310, or in other non-volatile storage devices for later execution.

[0093] AV architecture

[0094] Figure 4 Showing for AV (e.g., Figure 1 The example architecture 400 of the vehicle 100 shown is illustrated. Architecture 400 includes a sensing system 402 (sometimes referred to as a sensing circuit), a planning system 404 (sometimes referred to as a planning circuit), a control system 406 (sometimes referred to as a control circuit), a positioning system 408 (sometimes referred to as a positioning circuit), and a database system 410 (sometimes referred to as a database circuit). Each system plays a role in the operation of the vehicle 100. Commonly, systems 402, 404, 406, 408, and 410 can be... Figure 1This is a portion of the AV system 120 shown. In some embodiments, any of systems 402, 404, 406, 408, and 410 is a combination of computer software (e.g., executable code stored on a computer-readable medium) and computer hardware (e.g., one or more microprocessors, microcontrollers, application-specific integrated circuits (ASICs), hardware memory devices, other types of integrated circuits, other types of computer hardware, or any or all combinations of these hardware). Systems 402, 404, 406, 408, and 410 are each sometimes referred to as processing circuitry (e.g., computer hardware, computer software, or a combination of both). Any or all combinations of systems 402, 404, 406, 408, and 410 are also examples of processing circuitry.

[0095] In use, the planning system 404 receives data representing the destination 412 and determines data representing the trajectory 414 (sometimes called a route) that the vehicle 100 can travel to reach (e.g., arrive at) the destination 412. In order for the planning system 404 to determine the data representing the trajectory 414, the planning system 404 receives data from the sensing system 402, the positioning system 408, and the database system 410.

[0096] The sensing system 402 uses, for example, as Figure 1 One or more sensors 121 are shown to identify nearby physical objects. The objects are classified (e.g., grouped into types such as pedestrians, bicycles, cars, traffic signs, etc.), and a scene description including the classified objects 416 is provided to the planning system 404.

[0097] The planning system 404 also receives data representing the location 418 of the AV from the positioning system 408. The positioning system 408 determines the location of the AV by using data from sensor 121 and data (e.g., geographic data) from database system 410 to calculate the location. For example, the positioning system 408 uses data from GNSS (Global Navigation Satellite System) sensors and geographic data to calculate the longitude and latitude of the AV. In embodiments, the data used by the positioning system 408 includes high-precision maps with lane geometry properties, maps describing road network connectivity properties, maps describing lane physical properties (such as traffic speed, traffic volume, number of vehicle and bicycle lanes, lane width, lane traffic direction, or lane marking type and location, or combinations thereof), and maps describing the spatial locations of road features (such as intersections, traffic signs, or various types of other traffic signals). In embodiments, the high-precision map is constructed by adding data to a low-precision map via automatic or manual annotation.

[0098] The control system 406 receives data representing trajectory 414 and data representing AV position 418, and operates the AV control functions 420a-420c (e.g., steering, throttle, braking, ignition) in a manner that will cause the vehicle 100 to travel along trajectory 414 to reach destination 412. For example, if trajectory 414 includes a left turn, the control system 406 will operate the control functions 420a-420c in such a way that the steering angle of the steering function will cause the vehicle 100 to turn left, and the throttle and brake will cause the vehicle 100 to stop and wait for passing pedestrians or vehicles before making the turn.

[0099] AV input

[0100] Figure 5 The sensing system 402 is shown. Figure 4 The inputs used are 502a-502d (e.g., Figure 1 Examples of sensor 121 and outputs 504a-504d (e.g., sensor data) are shown. One input 502a is a LIDAR (light detection and ranging) system (e.g., Figure 1 The LIDAR system shown is 123. LIDAR is a technique that uses light (e.g., a beam of light such as infrared light) to obtain data related to physical objects in its line of sight. A LIDAR system produces LIDAR data as output 504a. For example, LIDAR data is a collection of 3D or 2D points (also called point clouds) used to construct a representation of environment 190.

[0101] Another input 502b is a RADAR (radar) system. RADAR is a technology that uses radio waves to acquire data related to nearby physical objects. RADAR can acquire data related to objects that are not within the line of sight of a radar system. The RADAR system generates RADAR data as output 504b. For example, RADAR data is one or more radio frequency electromagnetic signals used to construct a representation of the environment 190.

[0102] Another input 502c is a camera system. The camera system uses one or more cameras (e.g., a digital camera using a light sensor such as a charge-coupled device [CCD]) to acquire information about nearby physical objects. The camera system produces camera data as output 504c. Camera data is typically in the form of image data (e.g., data in image data formats such as RAW, JPEG, PNG, etc.). In some examples, the camera system has multiple independent cameras, for example, for stereoscopic imaging (stereoscopic vision), which enables the camera system to perceive depth. Although the objects perceived by the camera system are described here as "nearby," this is relative to the AV. In some embodiments, the camera system is configured to "see" distant objects (e.g., objects as far as 1 kilometer or more in front of the AV). Therefore, in some embodiments, the camera system has features such as sensors and lenses optimized for perceiving distant objects.

[0103] Another input 502d is a Traffic Light Detection (TLD) system. A TLD system uses one or more cameras to acquire information related to traffic lights, street signs, and other physical objects that provide visual navigation information. The TLD system produces TLD data as output 504d. TLD data is often in the form of image data (e.g., data in image data formats such as RAW, JPEG, PNG, etc.). The difference between a TLD system and a system that includes cameras is that a TLD system uses a camera with a wide field of view (e.g., using a wide-angle lens or fisheye lens) to acquire information related to as many physical objects as possible that provide visual navigation information, enabling the vehicle 100 to access all relevant navigation information provided by these objects. For example, the field of view of a TLD system is approximately 120 degrees or greater.

[0104] In some embodiments, sensor fusion technology is used to combine outputs 504a-504d. Thus, individual outputs 504a-504d are provided to other systems within the vehicle 100 (e.g., to systems such as...). Figure 4 The planning system 404 shown may provide combined outputs to other systems in the form of single or multiple combined outputs of the same type (e.g., using the same combination technique or combining the same outputs or both) or single or multiple combined outputs of different types (e.g., using different individual combination techniques or combining different individual outputs or both). In some embodiments, an early fusion technique is used. The early fusion technique is characterized by combining the outputs before applying one or more data processing steps to the combined outputs. In some embodiments, a late fusion technique is used. The late fusion technique is characterized by combining the outputs after applying one or more data processing steps to the individual outputs.

[0105] Figure 6 An example of a LIDAR system 602 is shown (e.g., Figure 5 The input 502a is shown. The LIDAR system 602 emits light 604a-604c from a emitter 606 (e.g., a laser emitter). The light emitted by the LIDAR system is typically not in the visible spectrum; for example, infrared light is often used. Some of the emitted light 604b encounters a physical object 608 (e.g., a vehicle) and is reflected back to the LIDAR system 602. (The light emitted from the LIDAR system typically does not penetrate the physical object, e.g., a solid physical object.) The LIDAR system 602 also has one or more photodetectors 610 for detecting the reflected light. In an embodiment, one or more data processing systems associated with the LIDAR system generate an image 612 representing the LIDAR system's field of view 614. Image 612 includes information representing the boundary 616 of the physical object 608. Thus, image 612 is used to determine the boundary 616 of one or more physical objects near the AV.

[0106] Figure 7 The diagram illustrates a LiDAR system 602 in operation. In the scenario shown, the vehicle 100 receives both camera system output 504c in the form of an image 702 and LiDAR system output 504a in the form of LiDAR data points 704. In use, the vehicle 100's data processing system compares the image 702 with the data points 704. Specifically, physical objects 706 identified in the image 702 are also identified in the data points 704. Thus, the vehicle 100 perceives the boundaries of physical objects based on the contours and density of the data points 704.

[0107] Figure 8 Additional details of the operation of the LIDAR system 602 are shown. As described above, the vehicle 100 detects the boundaries of physical objects based on the characteristics of the data points detected by the LIDAR system 602. Figure 8As shown, a flat object, such as ground 802, will reflect light 804a-804d emitted from LIDAR system 602 in a consistent manner. In other words, because LIDAR system 602 emits light at a consistent interval, ground 802 will reflect light back to LIDAR system 602 at the same consistent interval. When vehicle 100 is traveling on ground 802, if nothing obstructs the path, LIDAR system 602 will continue to detect light reflected by the next effective surface point 806. However, if object 808 obstructs the path, the light 804e-804f emitted by LIDAR system 602 will be reflected from points 810a-810b in a manner inconsistent with the expected consistency. Based on this information, vehicle 100 can determine the presence of object 808.

[0108] Path planning

[0109] Figure 9 Show (for example, as) Figure 4 The diagram 900 illustrates the relationship between the inputs and outputs of the planning system 404. Generally, the output of the planning system 404 is a route 902 from a starting point 904 (e.g., a source location or initial location) to an ending point 906 (e.g., a destination or final location). Route 902 is typically defined by one or more road segments. For example, a road segment refers to the distance to be traveled over at least a portion of a street, road, highway, driveway, or other physical area suitable for vehicle travel. In some examples, such as if the vehicle 100 is an off-road capable vehicle such as a four-wheel drive (4WD) or all-wheel drive (AWD) car, SUV, or pickup truck, route 902 includes “off-road” segments such as unpaved paths or open fields.

[0110] In addition to route 902, the planning system also outputs lane-level route planning data 908. Lane-level route planning data 908 is used to navigate segments of route 902 at specific times based on conditions. For example, if route 902 comprises a multi-lane highway, lane-level route planning data 908 includes trajectory planning data 910, which vehicle 100 can use to select a lane from the multiple lanes based on factors such as whether an exit is nearby, whether another vehicle is present in one or more of the multiple lanes, or other factors that change over a period of minutes or less. Similarly, in some implementations, lane-level route planning data 908 includes a speed constraint 912 specific to a segment of route 902. For example, if the segment includes pedestrians or unexpected traffic, speed constraint 912 can limit vehicle 100 to a slower speed than expected, such as a speed limit based on the segment's speed limit data.

[0111] In an embodiment, the inputs to the planning system 404 include (e.g., from...) Figure 4 The database system 410 shown contains database data 914 and current location data 916 (for example, Figure 4 The AV position shown is 418), (for example, for use with Figure 4 The destination data 918 and object data 920 shown for destination 412 (e.g., as shown) Figure 4 The perception system 402 shown perceives classified objects 416. In some embodiments, database data 914 includes rules used during planning. The rules are specified using a formal language (e.g., Boolean logic). At least some of these rules will apply to any given situation encountered by vehicle 100. A rule applies to a given situation if it has conditions satisfied based on information available to vehicle 100 (e.g., information about the surrounding environment). Rules can have priorities. For example, a rule “move to the leftmost lane if the road is a highway” can have a lower priority than “move to the rightmost lane if the exit is within a mile.”

[0112] Figure 10 This is illustrated in path planning (e.g., by planning system 404). Figure 4 The directed graph used is 1000. Generally speaking, such as... Figure 10 The directed graph 1000 shown is used to determine any path between a starting point 1002 and an ending point 1004. In the real world, the distance separating the starting point 1002 and the ending point 1004 may be relatively large (e.g., in two different urban areas) or relatively small (e.g., two intersections adjacent to a city block or two lanes of a multi-lane road).

[0113] In an embodiment, the directed graph 1000 has nodes 1006a-1006d representing different locations that a vehicle 100 may occupy between a starting point 1002 and an ending point 1004. In some examples, for instance, when the starting point 1002 and the ending point 1004 represent different urban areas, nodes 1006a-1006d represent road segments. In some examples, for instance, when the starting point 1002 and the ending point 1004 represent different locations on the same road, nodes 1006a-1006d represent different locations on that road. Thus, the directed graph 1000 includes information at different levels of granularity. In an embodiment, the directed graph with higher granularity is also a subgraph of another directed graph with a larger scale. For example, most of the information in a directed graph where the starting point 1002 and the ending point 1004 are far apart (e.g., many miles apart) is at a low granularity, and the directed graph is based on stored data, but the directed graph also includes some high-granularity information for representing a portion of the physical location in the field of view of the vehicle 100.

[0114] Nodes 1006a-1006d are distinct from objects 1008a-1008b that cannot overlap with nodes. In an embodiment, at a low granularity, objects 1008a-1008b represent areas that vehicles cannot pass through, such as areas without streets or roads. At a high granularity, objects 1008a-1008b represent physical objects within the field of view of vehicle 100, such as other vehicles, pedestrians, or other entities with which vehicle 100 cannot share physical space. In an embodiment, some or all of objects 1008a-1008b are static objects (e.g., objects that do not change position, such as streetlights or utility poles) or dynamic objects (e.g., objects capable of changing position, such as pedestrians or other cars).

[0115] Nodes 1006a-1006d are connected by edges 1010a-1010c. If two nodes 1006a-1006b are connected by edge 1010a, then vehicle 100 can travel between one node 1006a and the other node 1006b, for example, without having to travel to an intermediate node before reaching the other node 1006b. (When it is mentioned that vehicle 100 travels between nodes, it means that vehicle 100 travels between two physical locations represented by the respective nodes.) Edges 1010a-1010c are typically bidirectional, meaning that vehicle 100 can travel from a first node to a second node, or from a second node to a first node. In an embodiment, edges 1010a-1010c are unidirectional, meaning that vehicle 100 can travel from a first node to a second node, but not from a second node to a first node. When edges 1010a-1010c represent, for example, a one-way street, a single lane of a street, road, or highway, or other features that can only be traversed in one direction due to legal or physical constraints, edges 1010a-1010c are one-way.

[0116] In an embodiment, the planning system 404 uses a directed graph 1000 to identify a path 1012 consisting of nodes and edges between a start point 1002 and an end point 1004.

[0117] Edges 1010a-1010c have associated costs 1014a-1014b. Costs 1014a-1014b represent the resources that would be spent if vehicle 100 selected that edge. A typical resource is time. For example, if the physical distance represented by one edge 1010a is twice the physical distance represented by another edge 1010b, then the associated cost 1014a of the first edge 1010a can be twice the associated cost 1014b of the second edge 1010b. Other factors affecting time include anticipated traffic, the number of intersections, speed limits, etc. Another typical resource is fuel economy. Two edges 1010a-1010b can represent the same physical distance, but due to factors such as road conditions and anticipated weather, one edge 1010a may require more fuel than the other edge 1010b.

[0118] When the planning system 404 identifies the path 1012 between the starting point 1002 and the ending point 1004, the planning system 404 typically selects the path that is optimized for cost, such as the path that has the minimum total cost when the individual costs of the edges are added together.

[0119] AV control

[0120] Figure 11 Show (for example, as) Figure 4The diagram shows a block diagram 1100 of the inputs and outputs of the control system 406. The control system operates according to a controller 1102, which includes, for example, one or more processors similar to processor 304 (e.g., one or more computer processors such as microprocessors or microcontrollers or both); short-term and / or long-term data storage devices similar to main memory 306, ROM 308, and storage device 310 (e.g., memory area random access memory or flash memory or both); and instructions stored in the memory that, when executed (e.g., by one or more processors), perform the operation of controller 1102.

[0121] In one embodiment, controller 1102 receives data representing a desired output 1104. The desired output 1104 typically includes speed, such as rate and heading. The desired output 1104 may be based, for example, from (e.g., as...) Figure 4 The data received by the planning system 404 (as shown) is used as follows. Based on the desired output 1104, the controller 1102 generates data that can be used as throttle input 1106 and steering input 1108. Throttle input 1106 indicates the magnitude of the desired output 1104 by engaging the throttle of the vehicle 100 (e.g., acceleration control), for example, by engaging the steering pedal or another throttle control. In some examples, throttle input 1106 also includes data that can be used to engage the brakes of the vehicle 100 (e.g., deceleration control). Steering input 1108 indicates the steering angle, such as the steering control of the AV (e.g., steering wheel, steering angle actuator, or other function for controlling the steering angle), which should be positioned to achieve the desired output 1104.

[0122] In one embodiment, controller 1102 receives feedback used when adjusting inputs provided to throttle and steering. For example, if vehicle 100 encounters an obstacle 1110 such as a hill, the measured rate 1112 of vehicle 100 drops below the desired output rate. In another embodiment, any measured output 1114 is provided to controller 1102 so that necessary adjustments can be made, for example, based on the difference 1113 between the measured rate and the desired output. The measured output 1114 includes measured position 1116, measured speed 1118 (including rate and heading), measured acceleration 1120, and other outputs measurable by the sensors of vehicle 100.

[0123] In one embodiment, information related to the disturbance 1110 is detected in advance, for example, by a sensor such as a camera or a LiDAR sensor, and this information is provided to a predictive feedback system 1122. The predictive feedback system 1122 then provides information that the controller 1102 can use to make appropriate adjustments. For example, if the vehicle 100's sensors detect ("see") a hill, the controller 1102 can use this information to prepare to engage the throttle at an appropriate time to avoid significant deceleration.

[0124] Figure 12 A block diagram 1200 shows the inputs, outputs, and components of controller 1102. Controller 1102 has a rate analyzer 1202 that affects the operation of throttle / brake controller 1204. For example, the rate analyzer 1202 instructs throttle / brake controller 1204 to accelerate or decelerate using throttle / brake 1206 based on feedback received by, for example, controller 1102 and processed by the rate analyzer 1202.

[0125] The controller 1102 also has a lateral tracking controller 1208 that affects the operation of the steering wheel controller 1210. For example, the lateral tracking controller 1208 instructs the steering wheel controller 1210 to adjust the position of the steering angle actuator 1212 based on feedback received by the controller 1102 and processed by the lateral tracking controller 1208.

[0126] Controller 1102 receives several inputs for determining how to control the throttle / brake 1206 and the steering angle actuator 1212. Planning system 404 provides controller 1102 with information, for example, to select the heading of vehicle 100 at the start of operation and to determine which road segment vehicle 100 will cross when it reaches an intersection. Positioning system 408 provides controller 1102 with information describing the current location of vehicle 100, for example, so that controller 1102 can determine whether vehicle 100 is at the expected location based on the positive control of the throttle / brake 1206 and steering angle actuator 1212. In an embodiment, controller 1102 receives information from other inputs 1214, such as information received from a database, computer network, etc.

[0127] Ground modeling

[0128] Figure 13A This is an aerial view of scene 1300A, showing areas with obstructions. Figure 13A In the example, scenario 1300A is the environment at a specific time instance (e.g., Figure 1 A snapshot of the ground plane in environment 190). In scene 1300A, vehicle 1302 (e.g., Figure 1The vehicle 100 is shown as a visual representation with sensing within scene 1300A. This sensing can be performed by one or more sensors (e.g., Figure 1 The sensor 121 is used for this purpose. In the example, the sensor is a LIDAR (e.g., sensor 121). Figure 1 (LIDAR 123). In scene 1300, the potential point cloud data points are shown by multiple extent rings. These extent rings include extent rings 1304A, 1304B, 1304C, 1304D, 1304E, and 1304F. In the example, LiDAR-based ground height detection divides the point cloud into azimuth subintervals. Figure 13A In the example, three azimuth intervals, 1306A, 1306B, and 1306C, are shown using dashed lines. Each range ring represents a set of reliable data points in which ground elevation is directly measured within each azimuth interval.

[0129] Several techniques can be used to model the ground height around vehicle 1302. Traditionally, ground height detection is performed frame-by-frame. For example, instantaneous frame-by-frame ground height detection based on LiDAR first models the point cloud (e.g., Figure 5 The output 504a) is divided into azimuth intervals with predetermined ranges (e.g., each 1.5 degrees). Next, ground height is estimated independently within each azimuth interval based on the distance from points within that interval to the location in question. Traditionally, neighboring points within adjacent intervals are not considered. Ground height is estimated within each azimuth interval by grouping points together to collectively determine the ground error within that interval. An instantaneous ground height measurement at any distance includes (e.g., average, collective) the mean estimate of ground height, the mean estimate of slope, the uncertainty of ground height, the uncertainty of slope, or any combination thereof. In this example, the mean estimate and uncertainty are described via a Gaussian function. By dividing sensor data into azimuth intervals, the amount of computation required to calculate ground height measurements can be reduced. Sensors often capture large amounts of data, and grouping the data into azimuth intervals eliminates the time-consuming process of estimating ground height for each separate data point. Ground height is estimated independently within each azimuth interval based on points within that interval.

[0130] This technology models ground height as the height of the ground relative to the main frame of the vehicle. In an embodiment, the main frame of the vehicle has an origin located at the center of gravity of the vehicle and can accelerate, rotate, and translate relative to an inertial reference frame (e.g., a global map frame). Additionally, in an embodiment, the main frame of the vehicle has an origin located at the midpoint of the rear axle of the vehicle. In an embodiment, from a positioning system (e.g., Figure 4The positioning system 408) obtains the global map.

[0131] Instantaneous ground height detection initially calculates ground height estimates for the data points closest to the vehicle. In the example of scenario 1300A, for azimuth interval 1306A, ground height detection begins at range loop 1304A and detects ground height at the points corresponding to range loops 1304B and 1304C, respectively. In this embodiment, instantaneous ground height detection begins with the data points closest to the vehicle and continues to estimate ground height for data points farther from the vehicle until the limit of sensor data is encountered for each azimuth interval. In addition to estimating ground height, slope estimates are also provided.

[0132] In the example of scenario 1300A, vehicle 1308 is located within the sensor field of view of vehicle 1302. Azimuth interval 1306A is obstructed by vehicle 1308. Vehicle 1308 creates an occlusion area 1310 beyond vehicle 1308. The occluded area is a portion of the environment where direct sensor data is not available. In other words, the vehicle's sensors cannot capture data related to the occluded area. Generally, occlusion is a result of objects in the scene (e.g., natural obstacles 191, vehicles 193, pedestrians 192, cyclists, buildings, and other obstacles). In the example of scenario 1300A, vehicle 1308 prevents vehicle 1302 from obtaining sensor data from locations within area 1310. Similarly, wall 1312 prevents vehicle 1302 from obtaining sensor data corresponding to the occluded area 1314. Wall 1312 obstructs azimuth interval 1306B.

[0133] In the presence of obstructions, instantaneous measurements can return the mean estimated ground height, the mean estimated slope, the uncertainty of ground height, the uncertainty of slope, and any combination thereof at any distance from the vehicle. Ground height measurement under obstruction conditions is as follows: Figure 13B As shown, the uncertainty associated with ground height and slope is high when the area is obstructed. In the example, with obstruction, the uncertainty associated with ground height and slope exceeds the reliability threshold, rendering the measurements useless.

[0134] The azimuth interval 1306C includes a bounding box 1316. In this embodiment, the bounding box 1316 corresponds to a vehicle. Because the indicated ground point appears before the bounding box 1316, the bounding box 1316 does not cause any occlusion of the indicated ground point. As a result, no occluded area is formed within the azimuth interval 1306C. For example, consider an environment with a flat, level ground along an angle of incidence that increases with distance relative to the vehicle. Generally, a LiDAR can instantaneously detect ground height up to the range limits of the LiDAR. For example, the LiDAR is at least partially based on LiDAR hardware to capture point cloud data up to the range limits. Depending on the LiDAR model, type, and brand, the range limits associated with the LiDAR hardware may vary further. In this example, where point cloud data is available, the instantaneous ground height measurement is instantaneous ground height detection. Instantaneous ground height detection can be performed using the azimuth interval 1306C across the entire range of the sensor hardware. In the example, ground height is predicted for points outside the sensor range by continuing to propagate available ground height measurements along azimuth intervals. In this example, the ground heights outside the range limits are based on bounding boxes 1316 predicted by a LiDAR Semantic Network (LSN). In this embodiment, the bounding boxes 1316 predicted by the LSN are another form of ground height measurement fused using Bayesian filtering.

[0135] Figure 13A Scenario 1300A is not intended to indicate that Scenario 1300A will include Figure 13A All the components shown. Instead, scenario 1300A may include more than... Figure 13A Less ingredients or Figure 13A Additional components not shown (e.g., additional objects, occluded areas, etc.). Scene 1300A may include any number of additional components not shown, depending on the specific implementation details. Furthermore, instantaneous ground height detection, pseudo-ground height generation, and other described functions may be implemented partially or entirely in hardware and / or in a processor. For example, this function may be implemented using an application-specific integrated circuit, logic implemented in a processor, logic implemented in a dedicated graphics processing unit, or in any other device.

[0136] Figure 13B This is a diagram illustrating an example ground detection algorithm operating within an azimuth interval of 1300°B. The azimuth interval includes data from... Figure 13A The vehicle 1302 is shown with cell blocks 1306A, 1306B, and 1306C. Generally, azimuth cell block 1300B is shown as a hypothetical path through the azimuth cells along increasing distances from the vehicle, where obstructions occur (e.g., Figure 13A The cars 1308, 1312) and the range rings 1304A, 1304B, 1304C, 1304D, 1304E or 1304F. Additionally, see reference... Figure 13A The instantaneous ground height detection begins with the data point closest to vehicle 1302 and is available before obstruction occurs or the sensor hardware reaches its limits. For example, in azimuth interval 1306A, vehicle 1308 causes obstruction near range loop 1304D. In this embodiment, it is assumed that the area directly below vehicle 1308 is for accurate ground height detection.

[0137] Line 1320A represents a ground height measurement. In this example, the ground height is an instantaneous ground height measurement as described above. Dashed line 1322A represents the measurement uncertainty associated with the instantaneous ground height measurement. In this example, the measurement uncertainty is based on the reliability or accuracy of the instantaneous ground height measurement. Therefore, for the azimuth interval 1306A, the measurement uncertainty 1322A associated with the ground height measurement 1320A increases with increasing distance relative to the vehicle 1302 (assuming after obstruction). The measurement uncertainty decreases at the respective observed range loops 1304A, 1304B, and 1304C. Because instantaneous ground height detection typically uses the captured sensor data at each range loop to estimate the ground height, the measurement uncertainty decreases at each observed range loop, where each range loop represents a set of reliable data points for directly measuring the ground height. Therefore, the ground height measurements at each range loop are more reliable and certain than estimates occurring at points outside the range loops. In this embodiment, for locations outside the range ring, ground height is estimated based on ground height measured at adjacent range rings. Vehicle 1308 represents obstruction within the azimuth interval 1306A. Beyond vehicle 1308, the measurement uncertainty 1322A associated with ground height measurement 1320A increases with distance and does not decrease. In this embodiment, due to the lack of available sensor data for ground height estimation, the measurement uncertainty associated with ground height measurement increases with distance and does not decrease.

[0138] Similarly, for the azimuth interval 1306B, the measurement uncertainty increases with increasing distance from the vehicle 1302, and decreases at each observed range ring 1304A, 1304B, and 1304C until the wall 1312 obstructs the view. Beyond the wall 1312, the measurement uncertainty 1322B associated with the ground height measurement 1320B increases with increasing distance from the vehicle 1302. For the azimuth interval 1306C, the measurement uncertainty increases with increasing distance from the vehicle 1302, and decreases at each observed range ring 1304A, 1304B, 1304C, 1304D, 1304E, and 1304F until the range limit of the sensor hardware is encountered. Beyond the range limit, LSN prediction of the ground height 1320C based on the bounding box 1316 occurs.

[0139] Figure 13B The azimuth sub-interval 1300B is not intended to indicate that the azimuth sub-interval 1300B will be configured as follows: Figure 13B As shown. Instead, the azimuth interval 130°B can include more than... Figure 13B Less or Figure 13B Additional components not shown (e.g., additional objects, occluded areas, etc.). The azimuth sub-interval 1300B may include any number of additional components not shown, depending on the specific implementation details. Furthermore, instantaneous ground height detection, pseudo-ground height generation, and other described functions may be implemented partially or entirely in hardware and / or a processor. For example, this function may be implemented using an application-specific integrated circuit, logic implemented in a processor, logic implemented in a dedicated graphics processing unit, or in any other device.

[0140] exist Figure 13A and Figure 13B In the example, ground height measurements are determined for each azimuth cell independently of other azimuth cells. For instance, when determining instantaneous ground height, azimuth cell 1306C is not used to determine ground height estimates for azimuth cells 1306A or 1306B. As a result, ground height measurements determined using azimuth cells may not include neighboring ground heights and are spatially inconsistent. In other words, using azimuth cells to determine ground height measurements does not consider lateral ground detection information. Furthermore, instantaneous ground height measurements using azimuth cells occur at a single timestamp. In this embodiment, instantaneous ground height measurements lack historical information carried from previous timestamps. Therefore, instantaneous ground height measurements using azimuth cells are frame-by-frame measurements and are temporally inconsistent.

[0141] This technique enables ground height measurement with context including information such as lateral proximity, lane information, and LSN bounding boxes (e.g., LSN bounding box 1316). Lateral proximity information includes, for example, information from adjacent azimuth intervals. Lane information includes lane smoothness and slope. For example, ground height on the same lane is generally smoother, and estimated or prior lane slope can help better propagate ground height estimates. In embodiments, lane information can be used to group LIDAR points together under the assumption that points falling on a single lane segment (for local segments in a local neighborhood) will share similar heights and have relatively smooth and constant slopes. By incorporating contextual information, ground height measurements according to this technique are spatially and temporally consistent. In some embodiments, this technique fuses ground height measurements with other ground detection techniques. In embodiments, this technique creates a framework for spatially and temporally consistent ground height estimation by fusing with other ground detection techniques and produces reliable ground height estimates beyond occlusion and beyond the sensor's ground extent.

[0142] A system for spatially and temporally consistent ground modeling using information fusion

[0143] Figure 14 This is a block diagram of system 1400 for spatially and temporally consistent ground modeling using information fusion. System 1400 includes multiple ground height representations, including but not limited to detectors, estimators, and measurements. A Bayesian filtering architecture 1402 fuses the varying ground height representations and outputs spatially and temporally consistent ground height measurements.

[0144] The Bayesian filtering architecture 1402 uses one or more Bayesian techniques to make inferences based on observed data, such as parameter estimation (e.g., approximating the posterior state distribution over multiple parameters given some observed data) and / or model comparison (e.g., comparing the outputs of a set of approximation algorithms). In an embodiment, the Bayesian filtering architecture outputs a final value based on one or more inputs. In an embodiment, the final ground height estimate output by the Bayesian filtering architecture propagates from the posterior state distribution at timetamp T to the prior state distribution at timetamp T+1. The Bayesian filtering architecture can assign at least one weight to one or more ground height representations based on the context or uncertainty associated with the ground height representation (e.g., uncertainty of other measurement inputs). In an example, the weights assigned by the Bayesian filtering architecture are based on the measurement uncertainty associated with the ground height representation. For example, due to the low measurement uncertainty associated with the instantaneous ground height detection at the range loop, as referenced... Figure 13A and Figure 13BThe instantaneous ground height detection at the aforementioned range loop was assigned a higher weight. In contrast, due to the high measurement uncertainty associated with instantaneous ground height prediction, the reference... Figure 13A and Figure 13B The instantaneous ground height predictions described in the occluded areas or beyond the range limits of the sensor are assigned lower weights.

[0145] Figure 14 Multiple sources of ground height representation are included, such as an instantaneous ground height detector 1406, a pseudo ground height generator 1408, and other ground height measurements 1410, including a ground elevation prior map 1412, an LSN bounding box 1414, and a neural network ground detection 1416. At least one ground height representation is input into a Bayesian filtering architecture, weighted, and fused with other ground height representations to generate a final ground height estimate. For ease of description, the output of the Bayesian filtering architecture is described as a ground height representation. However, this technique is applicable to any ground measurement, including ground slope.

[0146] Positioning system 1404 (e.g., Figure 4 The positioning system 408 provides input to the Bayesian filtering architecture 1402. The positioning system 1404 provides vehicle attitude information, including the vehicle's orientation and location, using data from at least one sensor and data from a database system. In this example, the positioning system 1404 uses data from GNSS sensors and geographic data to calculate the vehicle's longitude and latitude. In this example, in addition to the vehicle attitude described above, the data output by the positioning system 1404 also includes a high-precision map of lane geometry, a map describing road network connectivity, and a map describing lane physical properties, etc. Therefore, in this embodiment, the positioning system 1404 provides ego vehicle movement (i.e., positioning information) for ground altitude propagation in the Bayesian filtering architecture 1402.

[0147] The instantaneous ground height detector 1406 determines the ground height measurement. For example, as referenced... Figure 13A and Figure 13BThe instantaneous ground height detector 1406 determines ground height measurements. Therefore, the ground height measurements performed by the instantaneous ground height detector 1406 estimate ground height for data points based on corresponding azimuth intervals. In addition to using azimuth intervals to determine ground height measurements, the instantaneous ground height detector 1406 includes a pseudo ground height generator 1408. The pseudo ground height generator 1408 calculates pseudo ground height measurements based at least in part on instantaneous ground height detection or prediction from the instantaneous ground height detector. Generally, pseudo ground height measurements are calculated by propagating instantaneous ground height measurements from one or more nearest neighbors. In an embodiment, instantaneous ground height measurements from a closer range (i.e., anchor units) are used to generate pseudo ground height measurements for locations outside that range. Reference is made below. Figure 15A The generation of pseudo-ground height is further described.

[0148] Other ground elevation representations 1410 are provided as input to the Bayesian filter architecture 1402. In some embodiments, other ground elevation representations may not be available for the Bayesian filter architecture 1402 or may not be provided as input to the Bayesian filter architecture 1402. Specific other ground elevation representations are described for illustrative purposes. However, the technique is not limited to the specific other ground elevation representations described herein and may include arbitrary ground elevation representations from arbitrary sources. Other potential measurements include ground elevation prior maps 1412, LSN bounding boxes 1414, and neural network ground detections 1416.

[0149] A ground elevation prior map 1412 characterizes the ground surface based on its height and slope. In the example, the ground elevation prior map provides ground elevation for various locations on a global map, where the global map is a high-resolution map. This technique transforms the prior map locations and orientations to the main frame coordinate system to extract ground elevation estimates using the ground elevation prior map 1412. In an embodiment, a Bayesian filtering architecture assigns higher weights to ground elevation measurements based on the elevation map 1412. A neural network ground detection 1416 uses a neural network to detect ground elevation. In the example, the neural network takes a three-dimensional (3D) point cloud as input and outputs an estimate of ground elevation. The neural network can be trained using one or more inputs. Generally, the LSN receives LIDAR sensor data as input and processes the sensor data to detect objects (e.g., Figure 1The environment may contain natural obstacles 191, vehicles 193, pedestrians 192; cyclists and other obstacles) or no objects. LSN generates a set of bounding boxes for potential objects in 3D space, along with confidence scores for the presence of object class instances (e.g., cars, pedestrians, or bicycles) within the bounding boxes. Higher classification scores indicate a greater likelihood that the corresponding object class instance exists within the box. In an embodiment, the bounding boxes (e.g., LSN box 1414) include an estimate of the ground height. In an embodiment, one or more bounding boxes are used to extract the ground height estimate.

[0150] Referring again to the Bayesian filter architecture 1402, the ground height for multiple data points is determined. For example, a ground height raster map (e.g., raster map 1500A in Figure 15) is used to divide the scene into one or more cells. The Bayesian filter architecture 1402 outputs ground height estimates for each cell of the ground height raster map. Generally, the height estimates for each cell location are modeled (e.g., described, represented) as the height of the center location of each cell.

[0151] In the example, the input to the Bayesian filter architecture 1402 is a raster map with instantaneous ground height and instantaneous measurement uncertainty for each cell location determined by azimuth binning. It also includes actually detected ground data (e.g., Figure 5 The outputs are 504a-504d; Figure 13A and Figure 13B The units are distinguished from other units and are labeled as anchor units. In an embodiment, the estimated ground height from the azimuth inter-cell partitioning and the instantaneous measurement uncertainty are used as the detected ground height. In the example, it is assumed that the detected ground height is accurate for anchor units. Additional pseudo-measurements are generated for units where no ground data is detected (such as units between LiDAR layers, units behind obstructions, or units beyond the range limits of the sensor). These units are labeled as non-anchor units. In an embodiment, the additional measurements include pseudo-ground height, ground elevation prior maps, LSN boxes, and neural network ground estimates. For non-anchor units, the ground height measurement is a fusion of the additional measurements. The Bayesian filtering architecture 1402 iteratively propagates the ground height measurements and updates the ground height for a series of timestamps.

[0152] In the example, at the Bayesian filtering architecture, the posterior state distribution (e.g., ground height measurement, slope) is assigned a timestamp T. The posterior state distribution at time T-1 is propagated to the prior state distribution at the current timestamp (time T). This propagation transforms the posterior state distribution into the prior state distribution at the current timestamp. In the embodiment, localization is used to propagate the ground height estimate from the posterior state distribution at timestamp T-1 to the prior state distribution at timestamp T. Localization is used for propagation because the grid map of the ground height estimate has an origin within the vehicle frame (local frame). As the vehicle moves, the previous estimate moves accordingly in the grid map. When a new ground height representation is received at time T, Bayesian filtering is used to integrate the new measurement and produce the posterior state distribution at time T. In the embodiment, the Bayesian filtering architecture iteratively updates the ground height measurements for each grid cell. In an embodiment, the Bayesian filtering architecture 1402 uses a Kalman filter to iteratively propagate ground height and slope (e.g., state) over time in the presence of noise such as uncertainty.

[0153] The propagation from time T-1 to time T introduces processing noise associated with the propagation uncertainty. Ground altitude propagates with propagation uncertainty. The propagation uncertainty between timestamps is used to calculate the uncertainty of the prior state distribution. In this embodiment, the propagation uncertainty is distinct from the measurement uncertainty (which indicates the reliability of the confidence level associated with the ground altitude measurement itself). Additionally, in this embodiment, the propagation uncertainty works in conjunction with all other uncertainties within the Bayesian filtering architecture to produce an optimal estimate. Uncertainties include, for example, instantaneous measurement uncertainty, other measurement input uncertainties, pseudo-propagation uncertainty, pseudo-measurement uncertainty, Bayesian propagation uncertainty, and Bayesian final output uncertainty.

[0154] In an embodiment, propagation uncertainty represents the variance between the posterior ground height estimate at timetamp T-1 (e.g., the "should" location) and the prior ground height estimate at timetamp T (e.g., the predicted location). Propagation uncertainty directly affects the ground height estimates for all locations. In an embodiment, the measurement uncertainty and propagation uncertainty associated with the ground height estimate at a timetamp are used to calculate at least one weight. For example, the smaller the propagation uncertainty (noise introduced due to state propagation processing), the higher the weight applied to the posterior ground height from previous timetamps. In this way, the Bayesian filtering architecture will favor the posterior ground height from previous timetamps. The higher the propagation uncertainty, the lower the weight applied to the posterior ground height from previous timetamps. This makes the Bayesian filtering architecture favor the current ground height measurement for a cell at time T. In an embodiment, the Bayesian filtering architecture outputs ground height estimates and multiple uncertainties for each cell.

[0155] This technique propagates information from one timestamp to another to achieve time consistency. Propagation uncertainty can increase due to positioning errors. Even with precise ground elevation determination, propagation uncertainty increases in the presence of positioning errors. For example, when positioning is inaccurate, based on received positioning data, a certain degree of pitch can be assumed in the vehicle, while the actual pitch may differ slightly. This can occur when a vehicle travels over speed bumps. This technique propagates multiple forms of uncertainty along with the associated ground elevation over time, enabling accurate ground elevation estimation that ignores ground elevation estimates associated with large positioning errors. Propagation noise is reduced by improving positioning. Therefore, this technique adaptively relies on positioning data because the positioning data is more reliable.

[0156] Figure 14 System 1400 is not intended to indicate that system 1400 includes Figure 14 All components shown. Instead, system 1400 may include more than... Figure 14 Fewer components or Figure 14 Additional components not shown may be included (e.g., additional ground height detectors, additional ground height measurements, or no additional ground height measurements). Depending on the specific implementation details, system 1400 may include any number of additional components not shown. Furthermore, instantaneous ground height detection, pseudo-measurement generation, and other described functions may be implemented partially or entirely in hardware and / or in a processor. For example, this function may be implemented using an application-specific integrated circuit, logic implemented in a processor, logic implemented in a dedicated graphics processing unit, or in any other device.

[0157] Figure 15AThis is an illustration of a two-dimensional (2D) ground elevation grid map 1500A. In an embodiment, Figure 14 The Bayesian filtering architecture 1402 outputs the final ground height measurement for each cell of the ground height raster map. In the embodiment, as described above, the raster map 1500A implements pseudo ground height generation (e.g., Figure 14 (Pseudo-ground height generator 1408). Therefore, although a single ground height raster map is shown, this technique can combine one or more raster maps of one or more resolutions.

[0158] In the example, the 2D ground elevation raster map 1500A includes vehicles (e.g., Figure 1 The ground elevation of the vehicle (100) is estimated. The resolution of the raster map 1500A is predetermined. For example, the raster captures a local area of ​​200m × 200m with a resolution cell of 0.2m × 0.2m. Each cell represents a small area with a fixed location in the local map, the origin of which is the vehicle frame. In this embodiment, the raster map is a sub-region from the global map. The ground elevation is estimated for each individual cell. The use of the raster map achieves a location-consistent canvas, ensuring that the location of the cells does not change.

[0159] The raster map 1500A includes anchor cells 1502 and non-anchor cells 1504. In an embodiment, an anchor cell is a cell with a direct ground point inside, or at least one cell with a direct ground point longitudinally located behind the anchor cell within the same azimuth interval. Cells with direct ground points longitudinally located behind the anchor cell are typically located between ground extent rings and are less likely to be occluded. In an embodiment, ground detection is good for anchor cells and is used as the ground height for that cell. In an embodiment, non-anchor cells, in contrast to anchor cells, represent cells that may be located behind occlusion or outside the range of LiDAR ground detection, and ground detection for non-anchor cells is less reliable. A pseudo ground height is calculated for each non-anchor cell.

[0160] For example, anchor cell 1502 is an area in a raster map where reliable ground elevation data exists. Figure 15AIn the example, the anchor cell is indicated by "A" in the cell. In the embodiment, reliable ground height data is a direct measurement of the ground height at a point within the cell. In the embodiment, reliable ground height data is an instantaneous ground height detection with low uncertainty derived from azimuth interval partitioning. In the example, the anchor cell is a cell containing a ground LiDAR point. In the example, the anchor cell is a cell that overlaps with an azimuth interval containing a ground LiDAR point. In the example, the anchor cell is a cell that is unlikely to be occluded by tall objects. The final ground height measurement output by the Bayesian filtering architecture 1402 is based on the detected ground LiDAR point or existing azimuth interval information associated with the cell.

[0161] In another example, non-anchor cells 1504 are cells without ground LiDAR points, or cells within azimuth intervals where no existing ground LiDAR points exist. Generally, non-anchor cells are cells in a raster map where reliable ground elevation data is unavailable. In this embodiment, non-anchor cells cannot be observed by direct sensor measurements and are more likely to be occluded. Figure 15A In the example, the non-anchor element 1504A may be due to occlusion. Occlusion can be, for example... Figure 13A The obscured area is 1310. In another example, the non-anchored units are outside the range of sensor hardware such as LiDAR. Non-anchored units 1504B and 1504C arise due to the range limitations of the sensor hardware. This technique determines the ground height estimate for the non-anchored units based at least in part on contextual data.

[0162] In the generation of pseudo-ground height measurements, the ground height estimate and the corresponding pseudo-measurement uncertainty and pseudo-propagation uncertainty are derived from the context data extracted from the raster map. Specifically, for the current non-anchor cell N, the nearest neighbor anchor cell A of the current non-anchor cell is found. i .exist Figure 15A In the example, the nearest neighbors are the eight nearest anchor cells along a 45° direction on the grid map 1500. In the embodiment, the nearest neighbors are the sixteen nearest anchor cells at every 22.5° along the grid map.

[0163] against Figure 15A For the non-anchor unit N, a total of eight arrows are drawn to illustrate the contextual data source used to generate the pseudo ground height for N. As shown in the figure, arrows 1506A, 1506B, 1506C, 1506D, 1506E, and 1506F each originate from an anchor unit where ground height estimation is available. In this embodiment, ground height estimates from each respective anchor unit are propagated, and a distance-based weighting factor is applied to compute the pseudo measurement for N.

[0164] exist Figure 15AIn the example, arrows 1506G and 1506H are shown with dashed lines. For each of arrows 1506G and 1506H, reliable anchor unit data is unavailable. In other words, the vehicle's sensor hardware cannot observe or obtain ground height estimates for these areas. In this embodiment, there is no nearest neighbor data contributing from the directions associated with arrows 1506G and 1506H. In this embodiment, prediction data may be obtained from the directions associated with arrows 1506G and 1506H. For example, when object prediction network data is available, a portion of the prediction data can be used to generate pseudo ground heights for non-anchor units N.

[0165] Figure 15A The raster map 1500A is not intended to indicate that the raster map 1500A will include Figure 15A All components shown. Instead, the raster map 1500A can include more than... Figure 15A Less ingredients or Figure 15A Additional components not shown (e.g., additional anchor cells, additional non-anchor cells, additional occluded areas, additional range-limited areas, etc.). Depending on the specific implementation details, the raster map 1500A may include any number of additional components not shown. Furthermore, instantaneous ground height detection, pseudo-measurement generation, and other described functions can be implemented partially or entirely in hardware and / or a processor. For example, this function can be implemented using an application-specific integrated circuit, logic implemented in a processor, logic implemented in a dedicated graphics processing unit, or in any other device.

[0166] Figure 15B This illustration was generated via pseudo-ground height measurement using a raster map 1500B. Figure 15B In the example, each nearest neighbor anchor cell provides context to help predict the ground height for the corresponding non-anchor cell. The predictions are combined to form a single pseudo-ground height measurement for non-anchor cell N 1504A. For non-anchor cell N 1504A, from one or more nearest neighbor anchor cells A i Generate a pseudo-ground height measurement. Anchor unit A i 1502A at its center of mass (X) bt ,Y bt It has a detected and reliable predicted ground height h i and slope h′ i Non-anchor element N and anchor element A i The distance between them is d = dist(N,A) i ), and anchor unit A i The yaw angle between the nominal diagonal direction and the non-anchor element N is Ψ. Based on anchor element A i The predicted pseudo-ground height of the non-anchor element N is:

[0167] z i =dh′ i cosΨ+h i

[0168] This assumes that N follows the anchor unit A. i The same slope, and the closest point P on the slope. i They have the same height at the same location. For pseudo-ground height measurement z i The uncertainty consists of two parts. The first part is P. i The pseudo-measurement uncertainty, and the second part is due to d from P i The pseudo-propagation uncertainty to N. Assume the pseudo-propagation uncertainty at a unit distance is... This can be obtained statistically from the data, and is related to anchor unit A. i ground height h i and slope h i The associated measurement uncertainty is called variance. and Covariance COV(h) i ,h′ i If the spurious measurement uncertainty is calculated as follows:

[0169]

[0170] Among them, COV(h) i ,h′ i This is the instantaneous measurement uncertainty. For ease of explanation, Figure 15B The previous example showed a pseudo-ground height prediction based on a single nearest neighbor anchor cell. However, this technique combines pseudo-ground height predictions from multiple anchor cells. The pseudo-ground heights from the nearest neighbors are combined to generate a single pseudo-measurement for non-anchor cells. The weighting factors for each nearest neighbor pseudo-measurement are:

[0171]

[0172] Where n is the number of neighboring anchor cells. The pseudo-ground height (e.g., mean) is calculated as a weighted arithmetic mean:

[0173]

[0174] The pseudo-measurement uncertainty (e.g., variance) of the combined pseudo-measurements is:

[0175]

[0176] Therefore, assume that the i-th nearest neighbor is A i And A i Ground height estimation is available, and this technique is derived from A iPredict / propagate pseudo-ground height / ground height to N. A i The greater the distance between the anchor and N, the more uncertain the prediction of the height of N becomes, and therefore the smaller the contribution of the pseudo-measurement to the combination of N. In summary, as described in the equations above, the mean (e.g., ground height) and variance (e.g., uncertainty) propagate from anchor units to non-anchor units. A single pseudo-measurement is formed for each non-anchor unit. Uncertainty exists for each anchor unit, and there is a distance between each anchor unit and non-anchor unit. In this embodiment, uncertainty propagates based on distance. A weighted average of all means (e.g., ground height) and variances (e.g., uncertainty) is calculated. In this embodiment, the weighting factor includes anchor unit A corresponding to the weighted arithmetic mean. i The uncertainty is the reciprocal of the value. Therefore, the greater the distance between anchor elements and non-anchor elements, the smaller the contribution of anchor elements to the pseudo-ground height measurement of non-anchor elements.

[0177] Figure 15B The raster map 1500B is not intended to indicate that the raster map 1500B will include Figure 15B All components shown. Instead, the raster map 1500B can include more than... Figure 15B Less ingredients or Figure 15B Additional components not shown (e.g., additional anchor cells, additional non-anchor cells, additional occluded areas, additional range-limited areas, etc.). Depending on the specific implementation details, the raster map 1500B may include any number of additional components not shown. Furthermore, pseudo-measurement generation and other described functions can be implemented partially or entirely in hardware and / or a processor. For example, this function can be implemented using an application-specific integrated circuit, logic implemented in a processor, logic implemented in a dedicated graphics processing unit, or in any other device.

[0178] Processing spatially and temporally consistent ground modeling using information fusion.

[0179] Figure 16A This is a flowchart of the processing 1600A for spatially and temporally consistent ground modeling using information fusion. At box 1602, the ground height for at least one anchor unit is obtained from instantaneous ground height estimation. Instantaneous ground height estimation includes, for example, instantaneous frame-by-frame ground height detection based on LiDAR. From the localization network (e.g., Figure 14 The positioning system 1404 obtains the vehicle's location and orientation. At box 1604, a pseudo-ground height measurement is calculated associated with a non-anchor cell of the vehicle's grid map. In this example, as described above, such as in reference... Figure 15A and Figure 15BThe method involves determining a pseudo ground height measurement. In some embodiments, at least one other ground height measurement is obtained. In an example, the at least one other ground height measurement is a ground elevation prior map, an LSN box, and a neural network ground detection, or any combination thereof. At box 1606, the ground height of the anchor unit and the non-anchor unit is estimated based on the ground height for the anchor unit from the instantaneous ground height detection, the pseudo ground height measurement calculated for each non-anchor unit, and the estimated ground height of the anchor unit and the non-anchor unit at the prior timestamp. In some embodiments, the ground height of the anchor unit and the non-anchor unit is estimated based on at least one other ground height measurement.

[0180] The Bayesian filtering architecture combines the ground heights of both anchor and non-anchor units, pseudo-ground height measurements, and the estimated ground height at time T-1 to produce a final ground height estimate at the current timestamp T. In some embodiments, when other ground height measurements are available (such as ground estimates from other models, such as ground elevation prior maps and LSN bounding boxes), the Bayesian filtering architecture combines the ground heights of both anchor and non-anchor units, pseudo-ground height measurements, other ground height measurements, and the estimated ground height at a previous timestamp (e.g., time T-1) to produce a final ground height estimate at the current timestamp (e.g., time T). At block 1608, the vehicle is operated based on the ground height estimate. In this embodiment, the final ground height estimate is output by the Bayesian filtering architecture for both anchor and non-anchor units. In some cases, the final ground height estimate for the anchor unit may differ from the initial ground height obtained from the instantaneous ground height estimate.

[0181] This process flowchart is not intended to indicate that the boxes in example process 1600A will be executed in any particular order, or that all boxes will be included in every case. Furthermore, depending on the specific implementation details, example process 1600A may include any number of additional boxes not shown.

[0182] Figure 16B This is a flowchart of process 1600B, which describes the process of spatially and temporally consistent ground modeling using information fusion at multiple timestamps. In box 1622, for the current timestamp T, the ground height for at least one anchor cell is obtained from the instantaneous ground height estimate. In box 1624, for the current timestamp T, pseudo-ground height measurements associated with non-anchor cells of the vehicle's raster map are calculated. In the example, as described above, as referenced... Figure 15A and Figure 15B The pseudo-ground height measurement is determined. The location and orientation of the vehicle are obtained at box 1623 and used to determine the pseudo-ground height measurement.

[0183] Box 1628A represents the final ground elevation estimate output by the Bayesian filtering architecture for a previous timestamp T-1. The positioning information at box 1623 is used to propagate the final ground elevation estimate at box 1628A for timestamp T-1 as the vehicle moves, so that the previous estimate is updated accordingly in the raster map as the vehicle moves. Therefore, the Bayesian filtering architecture iteratively updates the ground elevation measurements for each raster cell across various timestamps.

[0184] In some embodiments, at least one additional ground height measurement is obtained. Therefore, when the current timestamp T is available, one or more additional ground height measurements are obtained at box 1625. For example, at least one additional ground height measurement is a ground elevation prior map, an LSN bounding box, a neural network ground detection, or any combination thereof.

[0185] At box 1626, for the current timestamp T, the ground height of the anchor unit and non-anchor units is estimated based on the ground height for the anchor unit from the instantaneous ground height detection, the pseudo-ground height measurement calculated for each non-anchor unit, and the estimated ground heights of the anchor unit and non-anchor units at previous timestamps. Box 1628B represents the final ground height estimate output by the Bayesian filtering architecture at timestamp T. The final ground height estimates of the anchor unit and non-anchor units are propagated at timestamp T+1 based on updated positioning information corresponding to the vehicle's movement. The vehicle is then operated based on the final ground height estimate.

[0186] This process flowchart is not intended to indicate that the boxes in example process 1600B will be executed in any particular order, or that all boxes will be included in every case. Furthermore, depending on the specific implementation details, any number of additional boxes (not shown) may be included within example process 1600.

[0187] Figure 17 This is a flowchart of the process for determining the pseudo-ground height (1700). At box 1702, raster map data, including anchor cells and non-anchor cells, is obtained. In this example, the raster map data includes a raster map (e.g., ...). Figure 15A (Raster map 1500A). At frame 1704, based on the distance and yaw angle between the nearest anchor cell and the corresponding non-anchor cell (reference). Figure 15BThe process involves determining individual pseudo-ground heights, where each pseudo-ground height is associated with an uncertainty (e.g., pseudo-measurement uncertainty). At box 1706, a weighting factor is calculated based on the uncertainty. At box 1708, the weighting factor is applied to the individual pseudo-ground height measurement corresponding to the anchor unit. At box 1710, the individual pseudo-ground height measurements corresponding to multiple anchor units are combined to generate a final combined pseudo-ground height. In pseudo-measurement generation, ground height estimates and corresponding uncertainty measurements are derived from context data extracted from the nearest neighbor of the current non-anchor unit. At box 1712, the vehicle is navigated based on the pseudo-ground height estimate.

[0188] Figure 17 The processing flowchart is not intended to indicate that the boxes in example processing 1700 will be executed in any particular order, or that all boxes will be included in every case. Furthermore, depending on the specific implementation details, any number of additional boxes (not shown) may be included within example processing 1700.

[0189] In the preceding description, embodiments of the invention have been described with reference to numerous specific details, which may vary from implementation to implementation. Therefore, the specification and drawings should be considered illustrative rather than restrictive. The sole and exclusive indication of the scope of the invention, and what the applicant expects to be the scope of the invention, is the literal and equivalent scope of the claims published from this application in the specific form of the claims, including any subsequent amendments. Any definitions of terms expressly set forth herein for inclusion in such claims should be taken as meaning as such terms are used in the claims. Furthermore, when the term “comprising” is used in the preceding specification or appended claims, what follows that phrase may be an additional step or entity, or a sub-step / sub-entity of a previously stated step or entity.

Claims

1. A method for a vehicle, comprising: Use a positioning system to obtain the location and orientation of vehicles in the environment; Using at least one processor, an instantaneous ground height estimate of the anchor cell of the raster map for the vehicle is obtained based on the acquired vehicle location and orientation; The at least one processor is used to calculate the pseudo-ground height associated with the non-anchor cells of the raster map; Using a Bayesian filtering architecture, a final ground height estimate for the anchor unit and the non-anchor unit is generated based on the instantaneous ground height estimate for the anchor unit, the pseudo ground height associated with the non-anchor unit, the estimated ground height from a previous timestamp, or any combination thereof, wherein the anchor unit is a unit with actually detected ground data, and the non-anchor unit is a unit without detected ground data. as well as The vehicle is operated using a controller based on the final ground elevation estimate.

2. The method according to claim 1, comprising: At least one other ground height measurement is obtained using the at least one processor; as well as Using the Bayesian filtering architecture, a final ground height estimate for the anchor unit and the non-anchor unit is generated based on the instantaneous ground height estimate for the anchor unit, the pseudo ground height associated with the non-anchor unit, the estimated ground height from a previous timestamp, at least one other ground height measurement, or any combination thereof.

3. The method according to claim 1 or 2, wherein, The calculation of the pseudo-ground height for each non-anchor unit includes: Determine the number of anchor cells that are the nearest neighbor anchor cells associated with the non-anchor cells; and The pseudo-ground height of the non-anchor unit is determined based on a weighting factor and the ground height of the nearest neighbor anchor unit, wherein the weighting factor is associated with the distance of the nearest neighbor anchor unit relative to the non-anchor unit.

4. The method according to claim 3, wherein, The ground height of the nearest neighbor anchor unit is associated with the measurement uncertainty, which quantifies the confidence level associated with the corresponding ground height of the nearest neighbor anchor unit.

5. The method according to claim 4, wherein, The measurement uncertainty of the nearest neighbor anchor unit is used to determine the measurement uncertainty of the non-anchor unit.

6. The method according to claim 1 or 2, wherein, The Bayesian filtering architecture uses a Kalman filter to iteratively propagate over time, in the presence of uncertainty, the instantaneous ground height estimate for the anchor unit, the pseudo ground height associated with the non-anchor unit, the estimated ground height from a previous timestamp, and at least one other ground height measurement.

7. The method according to claim 2, wherein, The at least one other ground height measurement includes neural network ground height prediction, ground elevation prior map, LiDAR bounding box data, or any combination thereof.

8. The method according to claim 1 or 2, comprising: Using at least one processor, the slope associated with the anchor cell of the raster map of the vehicle is obtained based on the acquired vehicle location and orientation; The at least one processor is used to calculate the pseudo-slope associated with the non-anchor cells of the raster map; Using a Bayesian filtering architecture, the final slope for the anchor unit and the non-anchor unit is generated based on slope and pseudo-slope measurements; as well as The vehicle is operated using a controller based on the final slope for the anchor unit and the non-anchor unit.

9. The method according to claim 1 or 2, wherein, The final ground height estimates for the anchor unit and the non-anchor unit are consistent in both space and time.

10. A non-transitory computer-readable storage medium storing instructions that, when executed by at least one programmable computer processor of a vehicle, cause the at least one programmable computer processor to perform operations including: Obtain the location and orientation of vehicles in the environment; Based on the obtained vehicle location and orientation, an instantaneous ground height estimate of the anchor cell of the grid map for the vehicle is obtained; Calculate the pseudo-ground height associated with the non-anchor cells of the raster map; Based on the instantaneous ground height estimate for the anchor unit, the pseudo ground height associated with the non-anchor unit, the estimated ground height from a previous timestamp, or any combination thereof, a final ground height estimate for the anchor unit and the non-anchor unit is generated, wherein, The anchor unit is a unit with actual detected ground data, while the non-anchor unit is a unit without detected ground data. as well as The vehicle is operated based on the final ground elevation estimate.

11. The computer-readable storage medium of claim 10, comprising: Use at least one processor to obtain at least one other ground height measurement; as well as Using a Bayesian filtering architecture, a final ground height estimate for the anchor unit and the non-anchor unit is generated based on the instantaneous ground height estimate for the anchor unit, the pseudo ground height associated with the non-anchor unit, the estimated ground height from a previous timestamp, at least one other ground height measurement, or any combination thereof.

12. The computer-readable storage medium according to claim 10 or 11, wherein, The calculation of the pseudo-ground height for each non-anchor unit includes: Determine the number of anchor cells that are the nearest neighbor anchor cells associated with the non-anchor cells; and The pseudo-ground height of the non-anchor unit is determined based on a weighting factor and the ground height of the nearest neighbor anchor unit, wherein the weighting factor is associated with the distance of the nearest neighbor anchor unit relative to the non-anchor unit.

13. The computer-readable storage medium according to claim 12, wherein, The ground height of the nearest neighbor anchor unit is associated with the measurement uncertainty, which quantifies the confidence level associated with the corresponding ground height of the nearest neighbor anchor unit.

14. The computer-readable storage medium according to claim 13, wherein, The measurement uncertainty of the nearest neighbor anchor unit is used to determine the measurement uncertainty of the non-anchor unit.

15. The computer-readable storage medium according to claim 10 or 11, wherein, The Bayesian filtering architecture uses a Kalman filter to iteratively propagate over time, in the presence of uncertainty, the instantaneous ground height estimate for the anchor unit, the pseudo ground height associated with the non-anchor unit, the estimated ground height from a previous timestamp, and at least one other ground height measurement.

16. The computer-readable storage medium according to claim 10 or 11, comprising: Using at least one processor, the slope associated with the anchor cell of the raster map of the vehicle is obtained based on the acquired vehicle location and orientation; The at least one processor is used to calculate the pseudo-slope associated with the non-anchor cells of the raster map; Using a Bayesian filtering architecture, the final slope for the anchor unit and the non-anchor unit is generated based on slope and pseudo-slope measurements; as well as The vehicle is operated using a controller based on the final slope for the anchor unit and the non-anchor unit.

17. A computer program product comprising a program that causes a computer to perform the method according to any one of claims 1-9.

Images