Adaptive mapping for navigating autonomous vehicles in response to changes in the physical environment

Abstract

To provide autonomous vehicles and associated mechanical, electrical and electronic hardware, computer software and systems, and wired and wireless network communications, to provide map data for the autonomous vehicles.SOLUTION: A method may include: accessing subsets of multiple types of sensor data; aligning the subsets of sensor data relative to a global coordinate system based on the multiple types of sensor data to form aligned sensor data; and generating datasets of three-dimensional map data. The method further includes: detecting a change in data relative to at least two datasets of the three-dimensional map data; and applying the change in the data to form updated three-dimensional map data.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] Various embodiments relate generally to autonomous vehicles and associated mechanical, electrical, and electronic hardware, computer software and systems, and wired and wireless network communications that provide a fleet of autonomous vehicles as a service. More specifically, systems, devices, and methods are configured to provide updates locally (e.g., at the home location of the autonomous vehicles) or remotely, or both, to maps, such as three-dimensional (“3D”) maps, for navigating one or more of the vehicles adapted to changes in the environment through which the vehicles traverse. [Background technology]

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS This PCT international application is a continuation of U.S. application Ser. No. 14 / 932,963, entitled "AUTONOMOUS VEHICLE FLEET SERVICE AND SYSTEM," filed November 4, 2015, and U.S. application Ser. No. 14 / 932,959, entitled "AUTONOMOUS VEHICLE FLEET SERVICE AND SYSTEM," filed November 4, 2016, which is a continuation of U.S. application Ser. No. 14 / 932,966, entitled "TELEOPERATION SYSTEM AND METHOD FOR TRAJECTORY MODIFICATION OF AUTONOMOUS VEHICLES," filed November 4, 2015, and U.S. application Ser. No. 14 / 932,966, entitled "AUTOMATED EXTRACTION OF SEMANTIC INFORMATION TO ENHANCE INCREMENTAL MAPPING MODIFICATIONS FOR ROBOTIC VEHICLES," filed November 4, 2015. No. 14 / 932,940 entitled "COORDINATION OF DISPATCHING AND MAINTAINING FLEET OF AUTONOMOUS VEHICLES," filed November 4, 2015; U.S. patent application Ser. No. 14 / 756,995 entitled "COORDINATION OF DISPATCHING AND MAINTAINING FLEET OF AUTONOMOUS VEHICLES," filed November 4, 2015; U.S. patent application Ser. No. 14 / 756,992 entitled "ADAPTIVE AUTONOMOUS VEHICLE PLANNER LOGIC," filed November 4, 2015; U.S. patent application Ser. No. 14 / 756,991 entitled "SENSOR-BASED OBJECT-DETECTION OPTIMIZATION FOR AUTONOMOUS VEHICLES," filed November 4, 2015; and U.S. patent application Ser. No. 14 / 756,991 entitled "CALIBRATION FOR AUTONOMOUS VEHICLE" filed November 4, 2015. No. 14 / 756,996, entitled "A SYSTEM FOR IMPROVING HYBRID OPERATION," all of which are incorporated herein by reference in their entireties.

[0003] Various approaches to developing driverless vehicles primarily focus on automating conventional vehicles (e.g., manually driven automated vehicles) with the goal of producing driverless vehicles for consumer purchase. For example, some automobile companies and affiliates are modifying conventional automobiles and control mechanisms, such as steering, to provide consumers with the ability to own vehicles that can operate without a driver. In some approaches, conventional driverless vehicles perform safety-critical driving functions in some conditions but require the driver to assume control (e.g., steering, etc.) when the vehicle controller is unable to resolve certain challenges that could jeopardize the safety of the occupants.

[0004] While functional, conventional driverless vehicles generally have several drawbacks. For example, many driverless vehicles under development evolve from vehicles that require manual (i.e., human-controlled) steering and other similar automotive functions. Therefore, many driverless vehicles are based on the paradigm that a vehicle should be designed to accommodate a licensed driver with a specific seat or location reserved within the vehicle. As a result, driverless vehicles are suboptimally designed and generally miss opportunities to simplify vehicle design and conserve resources (e.g., reduce the cost of manufacturing driverless vehicles). Conventional driverless vehicles also have other drawbacks.

[0005] Other drawbacks exist in traditional transportation services, which are effectively poorly suited to managing vehicle inventory due to the general approach of providing traditional transportation and ride-sharing services, for example. In one traditional approach, passengers are required to access a mobile application to request transportation services through a centralized service that assigns a human driver and a vehicle (e.g., privately owned) to the passenger. By using vehicles owned by different owners, maintenance of the private vehicles and safety systems goes unchecked. In another traditional approach, some entity enables group vehicle pooling by allowing drivers who register as members to access vehicles shared among members. This approach is poorly suited to providing traditional transportation services because drivers must board and exit the shared vehicles at specific locations, which is rare in urban environments and requires access to relatively expensive real estate (i.e., parking lots) for parking the pooled vehicles. In the traditional approaches described above, traditional vehicles used to provide transportation services are typically underutilized from an inventory perspective because the vehicles are immobilized when the drivers leave. Additionally, ride-sharing schemes (and privately owned vehicle transportation services) are generally poorly adapted to rebalance inventory to match demand for transportation services to match usage and general driving patterns. Some previously described vehicles with limited self-driving automation are also poorly adapted to rebalance inventory because a human driver may generally be required. An example of a vehicle with limited self-driving automation is one designated as a Level 3 ("L3") vehicle by the U.S. Department of Transportation's National Highway Traffic Safety Administration ("NHTSA").

[0006] As another drawback, typical approaches to driverless vehicles are generally not well-adapted for detecting and navigating vehicles with respect to interactions (e.g., social interactions) between the moving vehicle and other vehicle operators or individuals. For example, some conventional approaches are not sufficiently capable of identifying pedestrians, bicyclists, etc., and associated interactions such as eye contact, gestures, etc., for purposes of addressing safety risks to driverless vehicle occupants and drivers of other vehicles, pedestrians, etc.

[0007] Therefore, what is needed is a solution to facilitate the implementation of autonomous vehicles without the limitations of conventional techniques. [Brief explanation of the drawings]

[0008] Various embodiments or examples (“Examples”) of the present invention are disclosed in the following detailed description and accompanying drawings.

[0009] [Figure 1] FIG. 1 illustrates an implementation of a fleet of autonomous vehicles communicatively networked to an autonomous vehicle service platform, according to some embodiments. [Figure 2] 1 is an example of a flow diagram for monitoring a fleet of autonomous vehicles, according to some embodiments. [Figure 3A] FIG. 1 illustrates examples of sensors and other autonomous vehicle components, according to some examples. [Figure 3B] FIG. 1 illustrates an example of sensor field redundancy and autonomous vehicle adaptation to loss of a sensor field, according to some examples. [Figure 3C] FIG. 1 illustrates an example of sensor field redundancy and autonomous vehicle adaptation to loss of a sensor field, according to some examples. [Figure 3D] FIG. 1 illustrates an example of sensor field redundancy and autonomous vehicle adaptation to loss of a sensor field, according to some examples. [Figure 3E]FIG. 1 illustrates an example of sensor field redundancy and autonomous vehicle adaptation to loss of a sensor field, according to some examples. [Figure 4] FIG. 1 is a basic block diagram illustrating a system including an autonomous vehicle service platform communicatively coupled to an autonomous vehicle controller via a communication layer, according to some examples. [Figure 5] FIG. 1 is an example of a flow diagram for controlling an autonomous vehicle, according to some embodiments. [Figure 6] FIG. 1 illustrates an example architecture for an autonomous vehicle controller, according to some embodiments. [Figure 7] FIG. 1 illustrates an example of an autonomous vehicle service platform that implements redundant communication channels to maintain reliable communications with a fleet of autonomous vehicles, according to some embodiments. [Figure 8] FIG. 1 illustrates an example of a messaging application configured to exchange data between various applications, according to some embodiments. [Figure 9] 9 illustrates types of data for facilitating remote operations using the communication protocol described in FIG. 8, according to some examples. [Figure 10] FIG. 10 illustrates an example of a teleoperator interface that allows the teleoperator to influence path planning, according to some embodiments. [Figure 11] FIG. 1 illustrates an example of a planner configured to invoke remote operations, according to some examples. [Figure 12] 1 is an example of a flow diagram configured to control an autonomous vehicle, according to some embodiments. [Figure 13] FIG. 10 illustrates an example in which a planner can generate a trajectory, according to some examples. [Figure 14] FIG. 1 illustrates another example of an autonomous vehicle service platform, according to some embodiments. [Figure 15] FIG. 1 is an example of a flow diagram for controlling an autonomous vehicle, according to some embodiments. [Figure 16] FIG. 1 is a diagram of an example of an autonomous vehicle fleet manager implementing a fleet optimization manager, according to some examples. [Figure 17] FIG. 1 is an example of a flow diagram for managing a fleet of autonomous vehicles, according to some embodiments. [Figure 18] FIG. 1 illustrates an autonomous vehicle fleet manager implementing an autonomous vehicle communication link manager, according to some embodiments. [Figure 19] FIG. 1 is an example of a flow diagram for determining an action for an autonomous vehicle during an event, according to some embodiments. [Figure 20] FIG. 1 illustrates an example of a localizer, according to some embodiments. [Figure 21] 1 is an example of a flow diagram for generating local pose data based on aggregate sensor data, according to some embodiments. [Figure 22] FIG. 10 illustrates another example of a localizer, according to some embodiments. [Figure 23] FIG. 2 illustrates an example of a perception engine, according to some embodiments. [Figure 24] FIG. 1 is an example of a flow diagram for generating perception engine data, according to some embodiments. [Figure 25] FIG. 2 illustrates an example of a segmentation processor, according to some embodiments. [Figure 26A] FIG. 2 illustrates an example of an object tracker and classifier, according to various embodiments. [Figure 26B] FIG. 10 illustrates another example of an object tracker, according to at least some examples. [Figure 27] FIG. 1 is a diagram of an example front-end processor of a perception engine, according to some examples. [Figure 28]FIG. 1 illustrates a simulator configured to simulate an autonomous vehicle in a synthetic environment, according to some embodiments. [Figure 29] FIG. 1 is an example of a flow diagram for simulating various aspects of an autonomous vehicle, according to some embodiments. [Figure 30] 1 is an example of a flow diagram for generating map data, according to some embodiments. [Figure 31] FIG. 1 illustrates the architecture of a mapping engine, according to some embodiments. [Figure 32] FIG. 1 illustrates an autonomous vehicle application, according to some embodiments. [Figure 33] 1A-1C illustrate examples of various computing platforms configured to provide various functions to components of an autonomous vehicle service, according to various embodiments. [Figure 34] 1A-1C illustrate examples of various computing platforms configured to provide various functions to components of an autonomous vehicle service, according to various embodiments. [Figure 35] 1A-1C illustrate examples of various computing platforms configured to provide various functions to components of an autonomous vehicle service, according to various embodiments. [Figure 36] FIG. 1 illustrates a mapping engine configured to adaptively generate mapping data for an autonomous vehicle in response to changes in a physical environment, according to some examples. [Figure 37] FIG. 1 illustrates an example of an autonomous vehicle controller implementing updated map data, according to some examples. [Figure 38] FIG. 10 is a flow diagram illustrating an example of generating map data, according to some examples. [Figure 39] FIG. 1 illustrates an example of a localizer configured to implement map data and locally generated map data, according to some examples. [Figure 40]FIG. 10 illustrates an example of a localizer configured to change the transmission rate or amount of locally generated sensor and / or map data, according to some examples. [Figure 41] FIG. 10 is a flow diagram illustrating an example of using varying amounts of locally generated map data to locate an autonomous vehicle, according to some examples. [Figure 42] 1A-1C illustrate examples of various computing platforms configured to provide various mapping-related functionality to components of an autonomous vehicle service, according to various embodiments. [Figure 43] 1A-1C illustrate examples of various computing platforms configured to provide various mapping-related functionality to components of an autonomous vehicle service, according to various embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0010] Various embodiments or examples may be implemented in several ways, including as a system, a process, a device, a user interface, a series of program instructions on a computer-readable medium, such as a computer-readable storage medium or a computer network through which the program instructions are transmitted via an optical, electronic, or wireless communications link. In general, the operations of the disclosed processes may be performed in any order unless otherwise provided in the claims.

[0011] A detailed description of one or more examples is provided below along with the accompanying drawings. The detailed description is provided in connection with such examples, but is not limited to any particular examples. Its scope is limited only by the claims and any alternatives, modifications, and equivalents thereof. In order to provide a thorough understanding, some specific details are set forth in the following description. These details are provided for purposes of example, and the described techniques can be practiced according to the claims without some or all of these specific details. For clarity, technical material known in the technical fields related to the examples has not been described in detail to avoid unnecessarily obscuring the description.

[0012] 1 is a diagram illustrating an implementation of a fleet of autonomous vehicles communicatively networked to an autonomous vehicle service platform, according to some embodiments. Diagram 100 illustrates a fleet of autonomous vehicles 109 (e.g., one or more of autonomous vehicles 109a-109e) operating as a service, with each autonomous vehicle 109 configured to automatically navigate a road network 110 and establish a communications link 192 with the autonomous vehicle service platform 101. In an example where the fleet of autonomous vehicles 109 constitutes a service, a user 102 can send a request 103 for autonomous transportation to the autonomous vehicle service platform 101 via one or more networks 106. In response, the autonomous vehicle service platform 101 can dispatch one of the autonomous vehicles 109 to autonomously transport the user 102 from a geographic location 119 to a geographic location 111. The autonomous vehicle service platform 101 may dispatch an autonomous vehicle from a station 190 to a geographic location 119, or may divert an autonomous vehicle 109c already in motion (e.g., without a passenger) to service a transportation request of a user 102. The autonomous vehicle service platform 101 may be further configured to divert an autonomous vehicle 109c in motion with a passenger in response to a request from a user 102 (e.g., as a passenger). Additionally, the autonomous vehicle service platform 101 may be configured to reserve an autonomous vehicle 109c in motion with a passenger for diversion to service a request of a user 102 after the existing passenger disembarks. Multiple autonomous vehicle service platforms 101 (not shown) and one or more stations 190 may be implemented to service one or more autonomous vehicles 109 in association with the road network 110. One or more stations 190 may be configured to store, service, manage, and / or maintain an inventory of autonomous vehicles 109 (e.g., stations 190 may include one or more computing devices that implement autonomous vehicle service platform 101).

[0013] According to some examples, at least some of the autonomous vehicles 109a-109e are configured as bidirectional autonomous vehicles, such as bidirectional autonomous vehicle (“AV”) 130. Bidirectional autonomous vehicle 130 may be configured to travel primarily in either direction along longitudinal axis 131, without limitation. Accordingly, bidirectional autonomous vehicle 130 may be configured to implement active lighting on the exterior of the vehicle to alert others in the immediate vicinity (e.g., other drivers, pedestrians, bicyclists, etc.) and the direction in which bidirectional autonomous vehicle 130 is traveling. For example, active light source 136 may be implemented as active light 138a when traveling in a first direction, or as active light 138b when traveling in a second direction. Active light 138a may be implemented using a first subset of one or more colors with optional animation (e.g., a light pattern of variable intensity light or a color that can change over time). Similarly, active light 138b may be implemented using a second subset of one or more colors and a light pattern that may differ from that of active light 138a. For example, active light 138a may be implemented using a white light as a “headlight,” while active light 138b may be implemented using a red light as a “taillight.” Active lights 138a and 138b, or portions thereof, may be configured to provide other color-related functions, such as providing a “turn signal indication” function (e.g., using a yellow light). According to various examples, logic within autonomous vehicle 130 may be configured to adapt active lights 138a and 138b to comply with various safety requirements and traffic regulations or laws for any number of jurisdictions.

[0014] In some embodiments, bidirectional autonomous vehicle 130 may be configured to have similar structural elements and components within each quadrant, such as quadrant 194. A quadrant, at least in this example, is shown as a portion of bidirectional autonomous vehicle 130 defined by the intersection of planes 132 and 134, both of which pass through the vehicle to form two similar halves on each side of planes 132 and 134. Additionally, bidirectional autonomous vehicle 130 may include an autonomous vehicle controller 147 that includes logic (e.g., hardware or software, or a combination thereof) configured to control several key vehicle functions, including driving controls (e.g., propulsion, steering, etc.) and active light sources 136, among other functions. Bidirectional autonomous vehicle 130 also includes several sensors 139 positioned at various locations on the vehicle (other sensors not shown).

[0015] Autonomous vehicle controller 147 may be further configured to determine the local attitude (e.g., local position) of autonomous vehicle 109 and detect external objects relative to the vehicle. For example, consider bidirectional autonomous vehicle 130 traveling in direction 199 within road network 110. A localizer (not shown) of autonomous vehicle controller 147 may determine a local position at geographic location 111. To do so, the localizer may use acquired sensor data, such as sensor data associated with the surfaces of buildings 115 and 117, which may be compared against reference data, such as map data (e.g., 3D map data including reflectance data), to determine the local position. Additionally, a perception engine (not shown) of autonomous vehicle controller 147 may be configured to detect, classify, and predict the behavior of external objects, such as external object 112 (“tree”) and external object 114 (“pedestrian”). Such external object classification may broadly categorize objects as static objects, such as external object 112, and dynamic objects, such as external object 114. The localizer and perception engine, as well as other components of the AV controller 147, work together to allow the autonomous vehicle 109 to drive autonomously.

[0016] According to some examples, the autonomous vehicle service platform 101 is configured to provide teleoperator services when an autonomous vehicle 109 requests teleoperation. For example, consider that an autonomous vehicle controller 147 in autonomous vehicle 109d detects an object 126 blocking path 124 on road 122 at point 191, as shown in inset 120. If the autonomous vehicle controller 147 cannot determine a path or trajectory along which vehicle 109d can safely travel with a relatively high degree of certainty, the autonomous vehicle controller 147 can send a request message 105 for teleoperation services. In response, teleoperator computing device 104 can receive instructions from teleoperator 108 to implement a maneuver to successfully (and safely) negotiate obstacle 126. Response data 107 can then be sent back to autonomous vehicle 109d to, for example, safely straddle a set of double lines when traveling along alternate path 121. In some examples, the teleoperator computing device 104 can generate a response that identifies geographic areas to exclude from path planning. In particular, rather than providing a path to follow, the teleoperator 108 can define areas or locations that the autonomous vehicle should avoid.

[0017] In view of the above, the structure and / or functionality of the autonomous vehicle 130 and / or autonomous vehicle controller 147, and their components, may perform real-time (or near real-time) trajectory calculations through autonomy-related operations such as localization and perception to enable the autonomous vehicle 109 to drive autonomously.

[0018] In some cases, the bidirectionality of the bidirectional autonomous vehicle 130 provides a vehicle with quadrants 194 (or any other number of symmetrical portions) that are similar or substantially similar to one another. Such symmetry reduces design complexity and, relative to one another, reduces the number of unique components or structures, thereby reducing inventory and manufacturing complexity. For example, the drivetrain and wheel system may be located in any of the quadrants. Additionally, the autonomous vehicle controller 147 is configured to invoke a teleoperation service to reduce the likelihood that the autonomous vehicle 109 will be delayed during travel while resolving events or issues that could otherwise affect occupant safety. In some cases, the visible portion of the road network 110 represents a geographically fenced area that can limit or otherwise control the movement of the autonomous vehicle 109 relative to the road network shown in FIG. 1 . According to various examples, the autonomous vehicle 109, and a fleet thereof, may be configurable to operate as a Level 4 (“fully self-driving” or L4) vehicle that can provide on-demand transportation with the convenience and privacy of point-to-point personal movement while offering the efficiency of a shared vehicle. In some examples, autonomous vehicle 109, or any autonomous vehicle described herein, may be configured to omit a steering wheel or any other mechanical means for providing manual (i.e., human-controlled) steering of autonomous vehicle 109. Additionally, autonomous vehicle 109, or any autonomous vehicle described herein, may be configured to omit a seat or location reserved within the vehicle for an occupant to engage with a steering wheel or any mechanical steering system.

[0019] FIG. 2 is an example flow diagram for monitoring a fleet of autonomous vehicles, according to some embodiments. Flow 200 begins at 202 when a fleet of autonomous vehicles is monitored. At least one autonomous vehicle includes an autonomous vehicle controller configured to autonomously move the vehicle from a first geographic area to a second geographic area. At 204, data representing an event associated with the vehicle's calculated confidence level is detected. An event may be a condition or situation that affects or may affect the operation of the autonomous vehicle. An event may be internal or external to the autonomous vehicle. For example, an obstacle blocking a road and reduced or lost communications may be viewed as an event. Events may include traffic conditions or congestion and an unexpected or unusual number or type of external objects (or trucks) perceived by the perception engine. Events may include weather-related conditions (e.g., loss of friction due to ice or rain) or a sun shining angle (e.g., at sunset), such as a low angle relative to the horizon that causes the sun to shine brightly on the eyes of human drivers of other vehicles. These and other conditions may be viewed as events that trigger a call to teleoperator service or cause the vehicle to execute a safe stopping trajectory.

[0020] At 206, data representing a subset of candidate trajectories may be received from the autonomous vehicle in response to detecting the event. For example, a planner in the autonomous vehicle controller may calculate and evaluate a large number of trajectories (e.g., thousands or more) per unit of time, such as one second. In some embodiments, the candidate trajectories are a subset of trajectories that provide a relatively higher confidence level that the autonomous vehicle can safely proceed forward given the event (e.g., using an alternate route provided by the teleoperator). Note that some candidate trajectories may be ranked higher or associated with a higher confidence level than other candidate trajectories. According to some examples, the subset of candidate trajectories may originate from any number of sources, such as a planner, a teleoperator computing device (e.g., a teleoperator may determine and provide an approximate route), etc., or may be combined as the subset of candidate trajectories. At 208, route guidance data may be identified in one or more processors. The route guidance data may be configured to assist the teleoperator in selecting a guided trajectory from one or more of the candidate trajectories. In some cases, the route guidance data specifies a value indicating a confidence level or probability that indicates the degree of certainty that a particular candidate trajectory can reduce or negate the possibility that an event may affect the operation of the autonomous vehicle. The guided trajectory as the selected candidate trajectory may be received at 210 in response to input from a teleoperator (e.g., the teleoperator may select at least one candidate trajectory as the guided trajectory from a group of otherwise ranked candidate trajectories). The selection may be made via an operator interface listing several candidate trajectories, for example, in order from highest confidence level to lowest confidence level. At 212, the selection of the candidate trajectory as the guided trajectory may be transmitted to the vehicle, which implements the guided trajectory to resolve the condition by causing the vehicle to perform a maneuver specified by the teleoperator. As such, the autonomous vehicle can transition out of the non-normative operating state.

[0021] 3A is a diagram illustrating example sensors and other autonomous vehicle components, according to some examples. Diagram 300 illustrates an interior view of a bidirectional autonomous vehicle 330, including sensors, a signal router 345, a drivetrain 349, a removable battery 343, an audio generator 344 (e.g., a speaker or transducer), and autonomous vehicle (“AV”) control logic 347. The sensors illustrated in diagram 300 include an image capture sensor 340 (e.g., any type of light capture device or camera), an audio capture sensor 342 (e.g., any type of microphone), a RADAR device 348, a sonar device 341 (or other similar sensors, including ultrasonic sensors or acoustic-related sensors), and a LIDAR device 346, among other sensor types and modalities (some of which, such as inertial measurement units or “IMUs,” global positioning system (“GPS”) sensors, sonar sensors, etc., are not shown). Note that quadrants 350 represent symmetry in each of the four “quadrant” sections of bidirectional autonomous vehicle 330 (e.g., each quadrant 350 may include wheels, drivetrain 349, similar steering mechanisms, similar structural supports and members, etc., in addition to those shown). As shown in FIG. 3A , similar sensors may be located in similar locations within each quadrant 350, although any other configuration may be implemented. Each wheel may be steerable individually and independently of one another. Also, removable batteries 343 may be configured to facilitate being swapped in and out rather than charged in place, thereby ensuring that downtime due to the need to charge batteries 343 is reduced or negligible. While autonomous vehicle controller 347a is shown as being used within bidirectional autonomous vehicle 330, autonomous vehicle controller 347a is not so limited and may be implemented within a unidirectional autonomous vehicle or any other type of vehicle, whether ground, air, or sea.It should be noted that the shown and described positions, locations, orientations, quantities, and types of sensors shown in FIG. 3A are not intended to be limiting, and as such, there may be any number and type of sensors, and any sensors may be positioned and oriented anywhere on the autonomous vehicle 330.

[0022] According to some embodiments, portions of the autonomous vehicle ("AV") control logic 347 may be implemented using a cluster of graphics processing units ("GPUs") that implements a framework and programming model suitable for programming a cluster of GPUs. For example, a programming language and application programming interface ("API") model compliant with the compute unified device architecture ("CUDA™") may be used to program the GPUs. CUDA™ is produced and maintained by NVIDIA of Santa Clara, California. It should be noted that other programming languages, such as OpenCL, or any other parallel programming language, may also be implemented.

[0023] According to some embodiments, autonomous vehicle control logic 347 may be implemented in hardware and / or software as autonomous vehicle controller 347a, which is shown to include motion controller 362, planner 364, perception engine 366, and localizer 368. As shown, autonomous vehicle controller 347a is configured to receive camera data 340a, lidar data 346a, and radar data 348a, or any other range sensing or localization data, including sonar data 341a, etc. Autonomous vehicle controller 347a is also configured to receive positioning data, such as GPS data 352, IMU data 354, and other position sensing data (e.g., wheel-related data such as steering angle, angular velocity, etc.). Additionally, autonomous vehicle controller 347a may receive any other sensor data 356 and reference data 339. In some cases, the reference data 339 includes map data (e.g., including 3D map data, 2D map data, 4D map data (e.g., Epoch Determination)) and route data (e.g., road network data including, but not limited to, RNDF data (or similar data), MDF data (or similar data)), and the like.

[0024] Localizer 368 is configured to receive sensor data from one or more sources, such as GPS data 352, wheel data, IMU data 354, lidar data 346a, camera data 340a, radar data 348a, etc., and reference data 339 (e.g., 3D map data and route data). Localizer 368 integrates (e.g., fuses) and analyzes the data by comparing the sensor data with map data to determine the local position (or location) of bidirectional autonomous vehicle 330. According to some embodiments, localizer 368 can generate or update the location or position of any autonomous vehicle in real time or near real time. Note that localizer 368 and its functionality need not be limited to “bidirectional” vehicles but may be implemented in any vehicle of any type. Therefore, localizer 368 (and other components of AV controller 347a) may be implemented in a “unidirectional” vehicle or any non-autonomous vehicle. According to some embodiments, the data describing the local position may include one or more of an x-coordinate, a y-coordinate, a z-coordinate (or any coordinate in any coordinate system, including polar or cylindrical coordinate systems, etc.), a yaw value, a roll value, a pitch value (e.g., an angular value), a rate (e.g., a velocity), an altitude, etc.

[0025] Perception engine 366 is configured to receive sensor data and local position data from one or more sources, such as lidar data 346a, camera data 340a, radar data 348a, etc. Perception engine 366 may be configured to determine the location of external objects based on the sensor data and other data. For example, external objects may be objects that are not part of a drivable surface. For example, perception engine 366 may be capable of detecting and classifying external objects as pedestrians, bicyclists, dogs, other vehicles, etc. (e.g., perception engine 366 is configured to classify objects according to a type of classification, which may be associated with semantic information including a label). Based on these external object classifications, the external objects may be labeled as dynamic or static objects. For example, external objects classified as trees may be labeled as static objects, while external objects classified as pedestrians may be labeled as dynamic objects. External objects labeled as static may or may not be described in the map data. Examples of external objects that may be labeled as static include a cement wall placed across a road, a lane closure sign, a newly placed mailbox, or a trash can adjacent to a road, etc. Examples of external objects that may be labeled as dynamic include bicyclists, pedestrians, animals, other vehicles, etc. If an external object is labeled as dynamic, further data about the external object may indicate a general level of activity and speed, as well as a behavior pattern, which is associated with a classification type. The further data about the external object may be generated by tracking the external object. As such, the classification type may be used to predict or otherwise determine the likelihood that the external object may interfere with, for example, an autonomous vehicle traveling along a planned path. For example, an external object classified as a pedestrian may be associated with some maximum and average speed (e.g., based on tracking data).The speed of the pedestrian relative to the speed of the autonomous vehicle can be used to determine whether a collision is likely. Additionally, the perception engine 364 can determine an uncertainty associated with the current and future states of the object. In some examples, the uncertainty may be expressed as an estimate (or probability).

[0026] Planner 364 is configured to receive perception data from perception engine 366 and may also include localizer data from localizer 368. According to some examples, perception data may include an obstacle map specifying static and dynamic objects located in the vicinity of the autonomous vehicle, while localizer data may include local positions or locations. In operation, planner 364 generates multiple trajectories and evaluates the trajectories based on at least the location of the autonomous vehicle relative to the relative locations of external dynamic and static objects. Planner 364 selects an optimal trajectory based on various criteria that guides the autonomous vehicle to provide a collision-free journey. In some examples, planner 364 may be configured to calculate the trajectory as a probabilistically determined trajectory. Additionally, planner 364 may send steering and propulsion commands (as well as deceleration or braking commands) to motion controller 362. The motion controller 362 can then convert any of the commands, such as steering commands, throttle or thrust commands, and braking commands, into control signals (e.g., for application to actuators or other mechanical interfaces) to effect changes in steering or wheel angle and / or velocity 353.

[0027] 3B-3E are diagrams illustrating examples of sensor field redundancy and autonomous vehicle adaptation to sensor field loss, according to some examples. Diagram 391 in FIG. 3B illustrates sensor field 301a in which sensor 310a detects objects (e.g., to determine range or distance or other information). While sensor 310a can implement any type of sensor or sensor modality, sensor 310a, as well as similarly described sensors such as sensors 310b, 310c, and 310d, may include lidar devices. Thus, sensor fields 301a, 301b, 301c, and 301d each include a laser-extended field. Diagram 392 in FIG. 3C illustrates four overlapping sensor fields, each generated by a corresponding lidar sensor 310 (not shown). As shown, portion 301 of the sensor field includes no overlapping sensor fields (e.g., a single lidar field), portion 302 of the sensor field includes two overlapping sensor fields, and portion 303 includes three overlapping sensor fields, thereby allowing such sensors to provide multiple levels of redundancy in the event that a lidar sensor fails.

[0028] FIG. 3D illustrates a loss of sensor field due to impaired operation of lidar 309, according to some examples. Sensor field 302 of FIG. 3C has been transformed into a single sensor field 305, one of sensor fields 301 of FIG. 3C has been lost to gap 304, and three of sensor fields 303 of FIG. 3C have been transformed into sensor field 306 (i.e., limited to two overlapping fields). When autonomous vehicle 330c is traveling in direction of travel 396, the sensor field in front of the moving autonomous vehicle may be less robust than one at its rear end. According to some examples, an autonomous vehicle controller (not shown) is configured to leverage the bidirectionality of autonomous vehicle 330c to address the loss of sensor field in the forward region in front of the vehicle. FIG. 3E illustrates bidirectional operation to restore some robustness to the sensor field in front of autonomous vehicle 330d. As shown, a more robust sensor field 302 coextensive with taillight 348 is located at the rear of vehicle 330d. When advantageous, autonomous vehicle 330d performs a bidirectional maneuver by pulling onto driveway 397 and switching its bidirectionality so that taillights 348 are actively switched to the other side (e.g., trailing edge) of autonomous vehicle 330d. As shown, autonomous vehicle 330d recovers a robust sensor field 302 in front of the vehicle as it travels along heading 398. Furthermore, the bidirectional maneuver described above obviates the need for more complicated maneuvers that require backing into a busy roadway.

[0029] 4 is a functional block diagram illustrating a system including an autonomous vehicle service platform communicatively coupled to an autonomous vehicle controller via a communication layer, according to some examples. Diagram 400 shows an autonomous vehicle controller (“AV”) 447 disposed within an autonomous vehicle 430, which includes several sensors 470 coupled to the autonomous vehicle controller 447. The sensors 470 include one or more lidar devices 472, one or more cameras 474, one or more radars 476, one or more global positioning system (“GPS”) data receiver-sensors, one or more inertial measurement units (“IMUs”) 475, one or more odometry sensors 477 (e.g., wheel encoder sensors, wheel speed sensors, etc.), and any other suitable sensors 478, such as infrared cameras or sensors, hyperspectral-capable sensors, ultrasonic sensors (or any other acoustic energy-based sensors), radio frequency-based sensors, etc. In some cases, wheel angle sensors configured to detect the steering angle of the wheels may be included as odometry sensors 477 or suitable sensors 478. In a non-limiting example, autonomous vehicle controller 447 may include four or more lidars 472, sixteen or more cameras 474, and four or more radar units 476. Further, sensors 470 may be configured to provide sensor data to components of autonomous vehicle controller 447 and elements of autonomous vehicle service platform 401. As shown in diagram 400, autonomous vehicle controller 447 includes a planner 464, a motion controller 462, a localizer 468, a perception engine 466, and a local map generator 440. Note that the elements shown in diagram 400 of FIG. 4 may include structure and / or functionality as similarly named elements described in connection with one or more other figures.

[0030] Localizer 468 is configured to locate (i.e., determine a local position) the autonomous vehicle relative to reference data, which may include map data, route data (e.g., road network data, such as RNDF-like data), and the like. In some instances, localizer 468 is configured to identify points in space that can represent, for example, the location of autonomous vehicle 430 relative to features of a representation of the environment. Localizer 468 is shown to include sensor data integrator 469, which can be configured to integrate multiple subsets of sensor data (e.g., of different sensor modalities) to reduce uncertainty associated with each individual type of sensor. According to some examples, sensor data integrator 469 is configured to fuse sensor data (e.g., lidar data, camera data, radar data, etc.) to form an integrated sensor data value for determining the local position. According to some examples, localizer 468 retrieves reference data from reference data repository 405, which may include map data repository 405a for storing 2D map data, 3D map data, 4D map data, etc. Localizer 468 may be configured to identify at least a subset of features in the environment to match against map data to identify or otherwise confirm the location of autonomous vehicle 430. According to some examples, localizer 468 may be configured to identify any amount of features in the environment, such that the set of features can include one or more features, or all features. In particular examples, any amount of lidar data (e.g., most or substantially all of the lidar data) may be compared to data representing a map for localization purposes. In general, inconsistent objects resulting from a comparison of environmental features to map data may be dynamic objects such as vehicles, bicyclists, pedestrians, etc. Note that detection of dynamic objects, including obstacles, may be performed with or without map data.In particular, dynamic objects may be detected and tracked independently of (i.e., in the absence of) map data. In some instances, 2D and 3D map data may be viewed as “global map data,” or map data that has been certified at a given time by autonomous vehicle service platform 401. Because the map data in map data repository 405a may be periodically updated and / or certified, there may be deviations between the map data and the actual environment in which the autonomous vehicle is positioned. Therefore, localizer 468 may retrieve locally derived map data generated by local map generator 440 to enhance localization. Local map generator 440 is configured to generate local map data in real time or near real time. Optionally, local map generator 440 may receive static and dynamic object map data to enhance the accuracy of the locally generated map, for example, by ignoring dynamic objects in localization. According to at least some embodiments, local map generator 440 may be integrated with or form part of localizer 468. In at least one instance, local map generator 440, either independently or in cooperation with localizer 468, may be configured to generate map and / or reference data based on simultaneous localization and mapping ("SLAM"), or the like. Note that localizer 468 may implement a "mixed" approach to map data usage, whereby logic within localizer 468 may be configured to select varying amounts of map data from either map data repository 405a or local map data from local map generator 440, depending on the reliability of each source of map data. Thus, localizer 468 may still use older map data in light of locally generated map data.

[0031] The perception engine 466 is configured to assist the planner 464 in planning routes and generating trajectories, for example, by identifying objects of interest in the surrounding environment through which the autonomous vehicle 430 is traveling. Additionally, a probability may be associated with each object of interest, whereby the probability may represent the likelihood that the object of interest may pose a threat to safe travel (e.g., a fast-moving motorcycle may require enhanced tracking than a person sitting on a bus stop bench reading a newspaper). As shown, the perception engine 466 includes an object detector 442 and an object classifier 444. The object detector 442 is configured to distinguish objects from other features in the environment, and the object classifier 444 can be configured to classify objects as either dynamic or static objects and track the location of the dynamic or static objects relative to the autonomous vehicle 430 for planning purposes. Additionally, the perception engine 466 can be configured to assign an identifier to a static or dynamic object that specifies whether the object is (or has the potential to be) an obstacle that may affect path planning in the planner 464. 4, it should be noted that the perception engine 466 can also perform other perception-related functions, such as segmentation and tracking, examples of which are described below.

[0032] Planner 464 is configured to generate several candidate trajectories for achieving the goal of reaching the destination via several available paths or routes. Trajectory evaluator 465 is configured to evaluate the candidate trajectories and identify which subset of the candidate trajectories is associated with the highest degree of confidence that the path to the destination will be collision-free. Trajectory evaluator 465 can then select an optimal trajectory based on relevant criteria that cause commands to generate control signals to vehicle components 450 (e.g., actuators or other mechanisms). Note that the relevant criteria may include any number of factors that define an optimal trajectory, and the selection need not be limited to reducing collisions. For example, trajectory selection may be made to optimize user experience (e.g., user comfort) and a collision-free trajectory that complies with traffic regulations and laws. User experience may be optimized by increasing or decreasing acceleration in various linear and angular directions (e.g., to reduce jerky running or other unpleasant movements). In some cases, at least some of the associated criteria may specify which of the criteria should take precedence or priority over other criteria while maintaining an optimized, collision-free journey. For example, legal constraints may be temporarily lifted or relaxed when generating a trajectory in limited circumstances (e.g., when crossing a double yellow line to accompany a bicyclist or traveling faster than the posted speed limit to match traffic flow). To that end, control signals are configured to cause propulsion and directional changes in the drivetrain and / or wheels. In this example, motion controller 462 is configured to translate commands into control signals (e.g., speed, wheel angle, etc.) for controlling the movement of autonomous vehicle 430. If trajectory evaluator 465 has insufficient information to ensure a sufficiently high confidence level to provide an optimized, collision-free journey, planner 464 may generate a request to teleoperator 404 for teleoperator assistance.

[0033] Autonomous vehicle service platform 401 includes teleoperator 404 (e.g., a teleoperator computing device), reference data repository 405, map updater 406, vehicle data controller 408, calibrator 409, and offline object classifier 410. Note that each element of autonomous vehicle service platform 401 may be independently located or distributed and can communicate with other elements in autonomous vehicle service platform 401. Furthermore, elements of autonomous vehicle service platform 401 can independently communicate with autonomous vehicle 430 via communication layer 402. Map updater 406 is configured to receive map data (e.g., from local map generator 440, sensors 460, or any other component of autonomous vehicle controller 447) and is further configured to detect deviations of map data in map data repository 405a from a locally generated map, for example. Vehicle data controller 408 can cause map updater 406 to update the reference data in repository 405 and facilitate updates to 2D, 3D, and / or 4D map data. In some cases, vehicle data controller 408 can control the rate at which local map data is received into autonomous vehicle service platform 408 and the frequency at which map updater 406 performs map data updates.

[0034] The calibrator 409 is configured to perform calibration of various sensors of the same or different types. The calibrator 409 may be configured to determine the relative position (e.g., in Cartesian space (x, y, z)) and orientation (e.g., roll, pitch, and yaw) of the sensors. The position and orientation of sensors such as cameras, lidar sensors, radar sensors, etc. may be calibrated with respect to other sensors and globally with respect to the vehicle's frame of reference. Offline self-calibration may also calibrate or estimate other parameters such as the vehicle inertia tensor, wheelbase, wheel radius, or road friction. Calibration may also be performed online to detect parameter changes, according to some examples. Note that calibration by the calibrator 409 may also include intrinsic parameters (e.g., optical distortion, beam angle, etc.) and extrinsic parameters of the sensors. In some cases, the calibrator 409 may be performed by, for example, maximizing the correlation between depth discontinuities in the 3D laser data and edges in the image data. Offline object classifier 410 is configured to receive data, such as sensor data, from sensors 470 or any other components of autonomous vehicle controller 447. According to some embodiments, the offline classification pipeline of offline object classifier 410 may be configured to pre-categorize and annotate objects (e.g., manually by a human and / or automatically using an offline labeling algorithm) and may be further configured to train an online classifier (e.g., object classifier 444) that can provide real-time classification of object types during online autonomous operation.

[0035] FIG. 5 is an example flow diagram for controlling an autonomous vehicle, according to some embodiments. At 502, flow 500 begins when sensor data from sensors of multiple modalities at the autonomous vehicle is received, for example, by an autonomous vehicle controller. One or more subsets of the sensor data may be combined to generate fused data, for example, to improve estimations. In some examples, at 504, sensor streams of one or more sensors (e.g., of the same or different modalities) may be fused to form fused sensor data. In some examples, subsets of lidar sensor data and camera sensor data may be fused at 504 to facilitate localization. At 506, data representing objects based on at least two subsets of sensor data may be derived in a processor. For example, data identifying static or dynamic objects may be derived (e.g., in a perception engine) from at least the lidar and camera data. At 508, detected objects that affect the planned path are determined, and at 510, a subset of trajectories is evaluated (e.g., in a planner) in response to the detected objects. At 512, a confidence level is determined that exceeds the range of acceptable confidence levels associated with normative behavior of the autonomous vehicle. Therefore, in this case, the confidence level may be such that the certainty of selecting the optimized route is less likely, whereby the optimized route promotes collision-free travel, complies with traffic laws, provides a pleasant user experience (e.g., a comfortable ride), and / or may be determined according to the probability of generating a candidate trajectory based on any other factors. Therefore, at 514, a request for an alternative route may be sent to the teleoperator computing device. The teleoperator computing device can then provide the planner with an optimal trajectory along which the autonomous vehicle should be driven. In some situations, the vehicle may also determine that a safety stop maneuver (e.g., safely and automatically stopping the autonomous vehicle in a location with a relatively low probability of hazard) is the best course of action.It should be noted that the order in which portions of a flow diagram are shown in this and other flow diagrams herein is not intended to imply a requirement to perform various functions in a linear manner, as portions may be performed sequentially, or in parallel with any one or more other portions of the flow diagram, and may be performed independently or dependent on other portions of the flow diagram.

[0036] FIG. 6 is a diagram illustrating an example of an autonomous vehicle controller architecture, according to some embodiments. Diagram 600 illustrates several processes, including a motion controller process 662, a planner process 664, a perception process 666, a mapping process 640, and a localization process 668, some of which may generate or receive data for other processes. Other processes, such as processes 670 and 650, may facilitate interaction with one or more mechanical components of the autonomous vehicle. For example, perception process 666, mapping process 640, and localization process 668 are configured to receive sensor data from sensor 670, while planner process 664 and perception process 666 are configured to receive guidance data 606, which may include route data such as road network data. Further to diagram 600, localization process 668 is configured to receive map data 605a (i.e., 2D map data), map data 605b (i.e., 3D map data), and local map data 642, among other types of map data. For example, localization process 668 can also receive other forms of map data, such as 4D map data, which may include, for example, epoch determinations. Localization process 668 is configured to generate local position data 641 representing a local position. Local position data 641 is provided to motion controller process 662, planner process 664, and perception process 666. Perception process 666 is configured to generate static and dynamic object map data 667, which may be transmitted to planner process 664. In some examples, static and dynamic object map data 667 may be transmitted along with other data, such as semantic classification information and predicted object behavior. Planner process 664 is configured to generate trajectory data 665 describing several trajectories generated by planner 664.The motion controller process uses the trajectory data 665 to generate low-level commands or control signals to apply to the actuators 650 to cause changes in steering angle and / or velocity.

[0037] FIG. 7 is a diagram illustrating an example of an autonomous vehicle service platform that implements redundant communication channels to maintain reliable communications with a fleet of autonomous vehicles, according to some embodiments. Diagram 700 illustrates an autonomous vehicle service platform 701 that includes a reference data generator 705, a vehicle data controller 702, an autonomous vehicle fleet manager 703, a teleoperator manager 707, a simulator 740, and a policy manager 742. The reference data generator 705 is configured to generate and modify map data and route data (e.g., RNDF data). Additionally, the reference data generator 705 may be configured to access 2D maps in a 2D map data repository 720, 3D maps in a 3D map data repository 722, and route data in a route data repository 724. Other map representation data and repositories, such as 4D map data including epoch determinations, may be implemented in some examples. The vehicle data controller 702 can be configured to perform various operations. For example, vehicle data controller 702 may be configured to vary the rate at which data is exchanged between the fleet of autonomous vehicles and platform 701 based on the quality level of communications over channel 770. During periods of bandwidth constraints, for example, data communications may be prioritized such that teleoperation requests from autonomous vehicles 730 are highly prioritized to ensure delivery. Furthermore, depending on the bandwidth available for a particular channel, variable levels of data abstraction may be transmitted per vehicle over channel 770. For example, in the presence of a robust network connection, all lidar data (e.g., substantially all lidar data, but may also be less) may be transmitted, while in the presence of a degraded or slow connection, a simpler or more abstract representation of the data (e.g., a bounding box with associated metadata, etc.) may be transmitted.The autonomous vehicle fleet manager 703 is configured to coordinate the dispatch of autonomous vehicles 730 to optimize multiple variables, including efficient use of battery power, run time, whether air conditioning in the autonomous vehicles 730 can be used during low battery states, etc., any or all of which variables may be monitored with consideration to optimizing a cost function associated with the operation of the autonomous vehicle service. Algorithms may be implemented to analyze various variables to minimize the cost or time of operation of the fleet of autonomous vehicles. Additionally, the autonomous vehicle fleet manager 703 maintains an inventory of autonomous vehicles and parts to adapt service schedules with consideration to maximizing fleet uptime.

[0038] The teleoperator manager 707 is configured to manage several teleoperator computing devices 704, to which a teleoperator 708 provides input. The simulator 740 is configured to simulate the operation of one or more autonomous vehicles 730 and the interaction between the teleoperator manager 707 and the autonomous vehicles 730. The simulator 740 can also simulate the operation of several sensors located within the autonomous vehicles 730 (including the introduction of simulated noise). Additionally, a city-like environment can be simulated such that a simulated autonomous vehicle can be introduced into a synthetic environment whereby simulated sensors can receive simulated sensor data, such as simulated laser returns. The simulator 740 can also provide other functions, including software updates and / or map data validation. The policy manager 742 is configured to maintain data representation policies or rules by which an autonomous vehicle should behave, taking into account various conditions or events the autonomous vehicle encounters while traveling on a road network. In some instances, updated policies and / or rules may be simulated in the simulator 740 to ensure safe operation of a fleet of autonomous vehicles in light of changes to the policies. Some of the above-described elements of autonomous vehicle service platform 701 are further described below.

[0039] Communication channel 770 is configured to provide a networked communication link between fleet of autonomous vehicles 730 and autonomous vehicle service platform 701. For example, communication channel 770 includes several different types of networks 771, 772, 773, and 774 with corresponding sub-networks (e.g., 771a-771n) to ensure a level of redundancy for reliable operation of the autonomous vehicle service. For example, the different types of networks within communication channel 770 may include different cellular network providers, different types of data networks, etc. to ensure sufficient bandwidth in the event of reduced or lost communication due to an outage of one or more of networks 771, 772, 773, and 774.

[0040] FIG. 8 is a diagram illustrating an example of messaging applications configured to exchange data between various applications, according to some embodiments. Diagram 800 illustrates a teleoperator application 801 located within a teleoperator manager and an autonomous vehicle application 830 located within an autonomous vehicle, whereby teleoperator application 801 and autonomous vehicle application 830 exchange message data via a protocol that facilitates communication over various networks, such as networks 871, 872, and other networks 873. According to some examples, the communication protocol is a middleware protocol implemented as Data Distribution Service™, with specifications maintained by the Object Management Group consortium. According to this communication protocol, teleoperator application 801 and autonomous vehicle application 830 can include a message router 854 located within a message domain, the message router configured to interface with teleoperator API 852. In some examples, message router 854 is a routing service. In some examples, message domain 850a in teleoperator application 801 may be identified by a teleoperator identifier, while message domain 850b may be identified as a domain associated with a vehicle identifier. Teleoperator API 852 in teleoperator application 801 is configured to interface with teleoperator processes 803a-803c, whereby teleoperator process 803b is associated with an autonomous vehicle identifier 804 and teleoperator process 803c is associated with an event identifier 806 (e.g., an identifier specifying an intersection that may be problematic for planning a collision-free path). Teleoperator API 852 in autonomous vehicle application 830 is configured to interface with autonomous vehicle operating system 840, which includes sensing application 842, perception application 844, localization application 846, and control application 848.In view of the above, the communication protocols described above can facilitate data exchange to facilitate teleoperation as described herein. Additionally, the communication protocols described above can be adapted to provide secure data exchange between one or more autonomous vehicles and one or more autonomous vehicle service platforms. For example, message router 854 can be configured to encrypt and decrypt messages to provide secure interaction between, for example, teleoperator process 803 and autonomous vehicle operating system 840.

[0041] FIG. 9 is a diagram illustrating types of data for facilitating teleoperation using the communication protocol described in FIG. 8 , according to some examples. Diagram 900 illustrates a teleoperator 908 interfacing with a teleoperator computing device 904 coupled to a teleoperator application 901 configured to exchange data via a data-centric messaging bus 972 implemented within one or more networks 971. The data-centric messaging bus 972 provides a communication link between the teleoperator application 901 and an autonomous vehicle application 930. A teleoperator API 962 of the teleoperator application 901 is configured to receive message service configuration data 964 and route data 960, such as road network data (e.g., data such as RNDF), mission data (e.g., MDF data), etc. Similarly, a messaging service bridge 932 is configured to receive messaging service configuration data 934. The messaging service configuration data 934 and 964 provide configuration data for configuring messaging services between the teleoperator application 901 and the autonomous vehicle application 930. Examples of messaging service configuration data 934 and 964 include quality of service ("QoS") configuration data implemented to configure a Data Distribution Service™ application.

[0042] An example of data exchange to facilitate teleoperation via a communications protocol is described as follows. Consider obstacle data 920 generated by the autonomous vehicle controller's perception system. Additionally, planner option data 924 is generated by the planner to inform the teleoperator of a subset of candidate trajectories, and position data 926 is generated by the localizer. The obstacle data 920, planner option data 924, and position data 926 are transmitted to a messaging service bridge 932, which generates telemetry data 940 and query data 942 according to message service configuration data 934, both of which are transmitted as telemetry data 950 and query data 952 via a data-centric messaging bus 972 to the teleoperator application 901. A teleoperator API 962 receives the telemetry data 950 and query data 952, which are processed taking into account route data 960 and message service configuration data 964. The resulting data is then presented to the teleoperators 908 via the teleoperator computing devices 904 and / or a collaborative display (e.g., a dashboard display visible to a group of collaborating teleoperators 908). The teleoperators 908 review the candidate trajectory options presented on the display of the teleoperator computing devices 904 and select a trajectory to be guided, which generates command data 982 and query response data 980, both of which are passed to the teleoperator API 962 as query response data 954 and command data 956. The query response data 954 and command data 956 are then transmitted to the autonomous vehicle application 930 as query response data 944 and command data 946 via a data-centric messaging bus 972. A messaging service bridge 932 receives the query response data 944 and command data 946 and generates teleoperator command data 928, which is configured to generate the teleoperator-selected trajectory for execution by the planner.It should be noted that the messaging process described above is not intended to be limiting and other messaging protocols may also be implemented.

[0043] FIG. 10 is a diagram illustrating an example of a teleoperator interface through which a teleoperator can influence path planning, according to some embodiments. Diagram 1000 illustrates an example of an autonomous vehicle 1030 in communication with an autonomous vehicle service platform 1001, including a teleoperator manager 1007 configured to facilitate teleoperation. In a first example, the teleoperator manager 1007 receives data requesting the teleoperator 1008 to look ahead at the autonomous vehicle's path approaching potential obstacles or areas of low planner confidence so that the teleoperator 1008 may be able to address challenges in advance. For illustrative purposes, consider that an intersection the autonomous vehicle is approaching may be tagged as problematic. Thus, the user interface 1010 displays a representation 1014 of the corresponding autonomous vehicle 1030 traveling along a path 1012, as predicted by several trajectories generated by the planner. Also displayed are other vehicles 1011 and dynamic objects 1013, such as pedestrians, that may cause sufficient confusion in the planner and thereby require teleoperator assistance. The user interface 1010 also presents the teleoperator 1008 with a current speed 1022, a speed limit 1024, and a current battery charge 1026. According to some examples, the user interface 1010 may display other data, such as sensor data, as obtained from the autonomous vehicle 1030. In a second example, the planner 1064 considers that several trajectories have been generated that are coextensive with the planner-generated path 1044, despite an unidentified object 1046 being detected. The planner 1064 may also generate a subset of candidate trajectories 1040, but in this example, the planner is unable to proceed given the current confidence level. If the planner 1064 is unable to determine an alternative path, a teleoperator request may be sent. In this case, the teleoperator can select one of the candidate trajectories 1040 to facilitate travel by the autonomous vehicle 1030 that is consistent with the teleoperator-based path 1042.

[0044] FIG. 11 illustrates an example planner configured to invoke teleoperation, according to some examples. Diagram 1100 illustrates a planner 1164 including a terrain manager 1110, a route manager 1112, a path generator 1114, a trajectory evaluator 1120, and a trajectory tracker 1128. The terrain manager 1110 is configured to receive map data, such as 3D map data or other similar map data, specifying topographical features. The terrain manager 1110 is further configured to identify candidate routes based on topography-related features on the route to the destination. According to various examples, the terrain manager 1110 receives 3D maps generated by sensors associated with one or more autonomous vehicles in the fleet. The route manager 1112 is configured to receive environmental data 1103, which may include traffic-related information associated with one or more routes that may be selected as routes to the destination. The route generator 1114 receives data from the terrain manager 1110 and the route manager 1112 and generates one or more routes or route segments suitable for guiding the autonomous vehicle toward the destination. Data representing one or more paths or path segments is sent to a trajectory evaluator 1120 .

[0045] The trajectory evaluator 1120 includes a state and event manager 1122, which may include a confidence level generator 1123. The trajectory evaluator 1120 further includes a guided trajectory generator 1126 and a trajectory generator 1124. Additionally, the planner 1164 is configured to receive policy data 1130, perception engine data 1132, and localizer data 1134.

[0046] Policy data 1130, according to some examples, can include criteria used by planner 1164 to determine a path with a sufficient confidence level for generating a trajectory. Examples of policy data 1130 include a policy specifying that trajectory generation is constrained by distance to external objects (e.g., maintaining a three-foot safety buffer from bicyclists whenever possible), or a policy requiring that trajectories not cross a center double yellow line, or a policy requiring that trajectories be limited to a single lane on a four-lane road (e.g., based on past events such as generally concentrating in the lane closest to a bus stop), and any other similar criteria specified by a policy. Perception engine data 1132 includes maps of locations of static and dynamic objects of interest, and localizer data 1134 includes at least the local position or locations.

[0047] The state and event manager 1122 can be configured to probabilistically determine an operating state of the autonomous vehicle. For example, a first operating state (i.e., “normative operation”) can describe a situation in which a trajectory is collision-free, while a second operating state (i.e., “non-normative operation”) can describe another situation in which a confidence level associated with a potential trajectory is insufficient to guarantee a collision-free run. According to some examples, the state and event manager 1122 is configured to use the perception data 1132 to determine an autonomous vehicle state that is normative or non-normative. The confidence level generator 1123 can be configured to analyze the perception data 1132 to determine the autonomous vehicle state. For example, the confidence level generator 1123 can use semantic information associated with static and dynamic objects and associated probability estimates to strengthen the certainty that the planner 1164 is determining a safe course of action. For example, the planner 1164 can use perception engine data 1132 that specifies the probability that an object is human or not human to determine whether the planner 1164 is operating safely (e.g., the planner 1164 can receive a degree of certainty that the object has a 98% probability that it is human and a 2% probability that it is not human).

[0048] Upon determining that the confidence level (e.g., based on statistical and probabilistic decisions) is below a threshold required for predicted safe operation, the relatively low confidence level (e.g., a single probability score) can trigger the planner 1164 to send a request 1135 for teleoperator assistance to the autonomous vehicle service platform 1101. In some cases, telemetry data and a set of candidate trajectories may accompany this request. Examples of telemetry data include sensor data, localization data, perception data, etc. The teleoperator 1108 can send the selected trajectory 1137 to the guided trajectory generator 1126 via the teleoperator computing device 1104. The selected trajectory 1137 is thus a trajectory formed with guidance from the teleoperator. Upon receiving confirmation that there is no change in state (e.g., a non-prescriptive state is pending), the guided trajectory generator 1126 passes the data to the trajectory generator 1124, which causes the trajectory tracker 1128, as a trajectory tracking controller, to use the teleoperator-specified trajectory to generate control signals 1170 (e.g., steering angle, speed, etc.). Note that the planner 1164 can trigger the transmission of a request 1135 for teleoperator assistance before the state transitions to a non-prescriptive state. In particular, the autonomous vehicle controller and / or its components can predict that a distant obstacle may be problematic and proactively cause the planner 1164 to invoke teleoperation before the autonomous vehicle reaches the obstacle. Otherwise, the autonomous vehicle can cause a delay by transitioning to a safe state (e.g., pulling over and stopping) following an encounter with an obstacle or scenario. In another example, teleoperation may be automatically invoked before the autonomous vehicle approaches a particular location known to be difficult to navigate. This decision may optionally take into account other factors, including time of day, position of the sun, if such conditions could cause disturbances to the reliability of sensor readings and traffic or accident data derived from various sources.

[0049] FIG. 12 is an example flow diagram configured to control an autonomous vehicle, according to some embodiments. Flow 1200 begins at 1202. Data representing a subset of objects is received at a planner within the autonomous vehicle, the subset of objects including at least one object associated with data representing a certainty of a classification type. For example, perception engine data can include metadata associated with the object, whereby the metadata specifies a certainty associated with a particular classification type. For example, a dynamic object can be classified as a "young pedestrian" with an 85% confidence level of accuracy. At 1204, localizer data can be received (e.g., at the planner). The localizer data can include map data generated locally within the autonomous vehicle. The local map data can specify a certainty (including uncertainty) that an event may occur in a certain geographic area. An event can be a condition or situation that affects or may affect operation of the autonomous vehicle. An event can be internal to the autonomous vehicle (e.g., a malfunctioning or impaired sensor) or external to the autonomous vehicle (e.g., a road blockage). Examples of events are described herein, such as in FIG. 2 and other figures and sections. A route coextensive with the geographic region of interest may be determined at 1206. For example, consider an event to be the positioning of the sun in the sky at a time when sunlight impairs a driver's visual function during rush-hour traffic conditions. Therefore, it is expected or predicted that traffic may slow down in response to bright sunlight. Thus, the planner may invoke teleoperation in advance if an alternative route to avoid the event is unlikely. At 1208, a local position is determined in the planner based on the local position data. At 1210, an operating state of the autonomous vehicle may be determined (e.g., probabilistically) based on, for example, the certainty of the classification type and the certainty of the event, which may be based on any number of factors such as speed, location, and other state information.To illustrate, consider an example in which a young pedestrian is detected by an autonomous vehicle during an event in which the vision of other drivers may be impaired by the sun, thereby creating an unsafe situation for the young pedestrian. Therefore, the relatively unsafe situation may be detected as a probabilistic event that may occur (i.e., an unsafe situation in which the teleoperator may be invoked). At 1212, a likelihood that the operating state is a normative state is determined, and based on this determination, a message is sent to the teleoperator computing device requesting the teleoperator to prepare in advance for a transition to the next operating state (e.g., prepare in advance for a transition from the normative operating state to a non-normative operating state, such as an unsafe operating state).

[0050] FIG. 13 is a diagram illustrating an example in which a planner can generate a trajectory, according to some examples. Diagram 1300 includes a trajectory evaluator 1320 and a trajectory generator 1324. Trajectory evaluator 1320 includes a confidence level generator 1322 and a teleoperator query manager 1329. As shown, trajectory evaluator 1320 is coupled to a perception engine 1366 to receive static map data 1301 and current and predicted object state data 1303. Trajectory evaluator 1320 also receives local position data 1305 from a localizer 1368 and plan data 1307 from a global planner 1369. In one operating state (e.g., non-normative), confidence level generator 1322 receives static map data 1301 and current and predicted object state data 1303. Based on this data, the confidence level generator 1322 may determine that the detected trajectory is associated with an unacceptable confidence level value, and therefore the confidence level generator 1322 sends the detected trajectory data 1309 (e.g., data including candidate trajectories) to notify the teleoperator via the teleoperator query manager 1329, which sends a request 1370 for teleoperator assistance.

[0051] In another operating state (e.g., a nominal state), static map data 1301, current and predicted object state data 1303, local position data 1305, and plan data 1307 (e.g., global plan data) are received by trajectory calculator 1325, which is configured to calculate trajectories (e.g., iteratively) to determine one or more optimal paths. At least one path is then selected and transmitted as selected path data 1311. According to some embodiments, trajectory calculator 1325 is configured to perform trajectory replanning, for example. Nominal driving trajectory generator 1327 is configured to generate trajectories in a sophisticated manner, such as by generating a trajectory based on a receding horizon control technique. Nominal driving trajectory generator 1327 can then transmit nominal driving trajectory path data 1372 to a trajectory tracker or vehicle controller for implementing physical changes, such as steering, acceleration, and other components.

[0052] 14 is a diagram illustrating another example of an autonomous vehicle service platform, according to some embodiments. Diagram 1400 illustrates an autonomous vehicle service platform 1401 including a teleoperator manager 1407 configured to manage interactions and / or communications between teleoperators 1408, teleoperator computing devices 1404, and other components of the autonomous vehicle service platform 1401. Further to diagram 1400, autonomous vehicle service platform 1401 includes a simulator 1440, a repository 1441, a policy manager 1442, a reference data updater 1438, a 2D map data repository 1420, a 3D map data repository 1422, and a route data repository 1424. Other map data, such as 4D map data (e.g., using epoch determination), may also be implemented and stored in a repository (not shown).

[0053] Teleoperator action recommendation controller 1412 includes logic configured to receive and / or control teleoperator service requests via autonomous vehicle (“AV”) planner data 1472, which may include requests for teleoperator assistance as well as telemetry and other data. As such, planner data 1472 may include recommended candidate trajectories or paths from which a teleoperator 1408 may select via teleoperator computing device 1404. According to some examples, teleoperator action recommendation controller 1412 may be configured to access other sources of recommended candidate trajectories from which to select an optimal trajectory. For example, the candidate trajectories included in autonomous vehicle planner data 1472 may be introduced in parallel into simulator 1440, which is configured to simulate events or conditions being experienced by an autonomous vehicle requesting teleoperator assistance. Simulator 1440 has access to map data and other data necessary to conduct simulations for a set of candidate trajectories, thereby avoiding the need for simulator 1440 to exhaustively repeat simulations to ensure sufficiency. Rather, the simulator 1440 may validate candidate trajectories or otherwise warn the teleoperator to be careful in their selections.

[0054] The teleoperator interaction capture analyzer 1416 can be configured to capture a large number of teleoperator transactions or interactions for storage in the repository 1441, which can, for example, in at least some instances, accumulate data related to several teleoperator transactions for analysis and generation of policies. According to some embodiments, the repository 1441 can also be configured to store policy data for access by the policy manager 1442. Additionally, the teleoperator interaction capture analyzer 1416 can apply machine learning techniques to empirically determine how best to respond to events or conditions that trigger a request for teleoperator assistance. In some instances, the policy manager 1442 can be configured to update a particular policy or generate a new policy in response to analysis of a large set of teleoperator interactions (e.g., after applying the machine learning techniques). The policy manager 1442 manages policies, which can be viewed as rules or guidelines upon which the autonomous vehicle controller and its components act to conform to the autonomous operation of the vehicle. In some cases, modified or updated policies may be applied to simulator 1440 to verify the effectiveness of permanently releasing or implementing such policy changes.

[0055] Simulator interface controller 1414 is configured to provide an interface between simulator 1440 and teleoperator computing device 1404. Consider, for example, that sensor data from a fleet of autonomous vehicles is applied to reference data updater 1438 via autonomous (“AV”) fleet data 1470, thereby configuring reference data updater 1438 to generate updated map and route data 1439. In some embodiments, updated map and route data 1439 can be released in advance as an update to the data in map data repositories 1420 and 1422 or as an update to the data in route data repository 1424. In this case, such data can be tagged as being a “beta version,” for example, where a lower threshold for requesting teleoperator service can be implemented when map tiles containing the previously updated information are used by an autonomous vehicle. Furthermore, updated map and route data 1439 can be introduced into simulator 1440 to validate the updated map data. Upon full release (e.g., at the end of beta testing), a previously lowered threshold for requesting teleoperator service is canceled. The user interface graphics controller 1410 provides rich graphics to the teleoperator 1408 so that a fleet of autonomous vehicles can be simulated within the simulator 1440 and accessed via the teleoperator computing device 1404 as if the simulated fleet of autonomous vehicles were real.

[0056] FIG. 15 is an example flow diagram for controlling an autonomous vehicle, according to some embodiments. Flow 1500 begins at 1502. Message data for managing a fleet of autonomous vehicles may be received at a teleoperator computing device. The message data may indicate event attributes associated with non-normative operating conditions in the context of a planned path for the autonomous vehicles. For example, the event may be characterized as a particular intersection becoming problematic due to a large number of pedestrians hastily crossing the street, violating traffic signals. The event attributes describe characteristics of the event, such as the number of people crossing the street, traffic delays resulting from the increased number of pedestrians, etc. At 1504, a teleoperation repository may be accessed to retrieve a first subset of recommendations based on simulated operation of aggregated data associated with the group of autonomous vehicles. In this case, the simulator may be a source of recommendations with which the teleoperator can implement. Additionally, the teleoperation repository may also be accessed to retrieve a second subset of recommendations based on an aggregation of teleoperator interactions in response to similar event attributes. In particular, the teleoperator interaction capture analyzer can apply machine learning techniques to empirically determine how best to respond to events with similar attributes based on previous requests for teleoperator assistance. At 1506, the first subset and the second subset of recommendations are combined to form a set of recommended strategies for the autonomous vehicle. At 1508, a representation of the set of recommended strategies can be visually presented on a display of the teleoperator computing device. At 1510, a data signal representing a selection (e.g., by the teleoperator) of the recommended strategy can be detected.

[0057] FIG. 16 is a diagram of an example autonomous vehicle fleet manager implementing a fleet optimization manager, according to some examples. Diagram 1600 shows an autonomous vehicle fleet manager configured to manage a fleet 1630 of autonomous vehicles traveling within a road network 1650. The autonomous vehicle fleet manager 1603 is coupled to a teleoperator 1608 via a teleoperator computing device 1604 and is also coupled to a fleet management data repository 1646. The autonomous vehicle fleet manager 1603 is configured to receive policy data 1602 and environmental data 1606, as well as other data. Further to diagram 1600, the fleet optimization manager 1620 is shown to include a movement request processor 1631, which includes a fleet data extractor 1632 and an autonomous vehicle dispatch optimization calculator 1634. The movement request processor 1631 is configured to process movement requests, such as those from a user 1688 requesting autonomous vehicle service. Fleet data extractor 1632 is configured to extract data related to the autonomous vehicles in the fleet. The data associated with each autonomous vehicle is stored in repository 1646. For example, the data about each vehicle can describe maintenance issues, scheduled service calls, daily usage, battery charge and discharge rates, and any other data that can be updated in real time and used for purposes of optimizing the fleet of autonomous vehicles to minimize downtime. Autonomous vehicle dispatch optimization calculator 1634 is configured to analyze the extracted data and calculate an optimized usage of the fleet to ensure that the next vehicle dispatched, such as from station 1652, provides the minimum travel time and / or cost within the fleet for autonomous vehicle service.

[0058] Fleet optimization manager 1620 is shown to include a mixed autonomous / non-autonomous vehicle processor 1640, which includes an AV / non-AV optimization calculator 1642 and a non-AV selector 1644. According to some examples, mixed autonomous / non-autonomous vehicle processor 1640 is configured to manage a mixed fleet of autonomous vehicles and vehicles driven by humans (e.g., as independent contractors). As such, autonomous vehicle services can utilize non-autonomous vehicles to meet excess demand or in areas, such as non-AV service area 1690, that may cross geographic fences or in areas of insufficient communication coverage. AV / non-AV optimization calculator 1642 is configured to optimize usage of the autonomous vehicle fleet and invite non-AV drivers into transportation services (e.g., with minimal or no disruption to the autonomous vehicle service). The non-AV selector 1644 includes logic for selecting the number of non-AV drivers to assist based on the calculations derived by the AV / non-AV optimization calculator 1642.

[0059] FIG. 17 is an example flow diagram for managing a fleet of autonomous vehicles, according to some embodiments. Flow 1700 starts at 1702. At 1702, policy data is received. The policy data may include parameters that specify how to best apply policies to select an autonomous vehicle to service the travel request. At 1704, fleet management data from a repository may be extracted. The fleet management data includes a subset of data regarding the pool of autonomous vehicles (e.g., data describing the readiness of a vehicle to service the transportation request). At 1706, data representing a travel request is received. For illustrative purposes, the travel request may be for transportation from a first geographic location to a second geographic location. At 1708, attributes based on the policy data are calculated to determine a subset of autonomous vehicles that are available to service the request. For example, the attributes may include battery charge level and time until next scheduled maintenance. At 1710, an autonomous vehicle is selected as transportation from the first geographic location to the second geographic location, and data is generated for dispatching the autonomous vehicle to a third geographic location associated with the origin of the travel request.

[0060] FIG. 18 is a diagram illustrating an autonomous vehicle fleet manager implementing an autonomous vehicle communication link manager, according to some embodiments. Diagram 1800 illustrates an autonomous vehicle fleet manager configured to manage a fleet of autonomous vehicles 1830 traveling within a road network 1850 corresponding to a communication outage identified as an “area of ​​reduced communications” 1880. The autonomous vehicle fleet manager 1803 is coupled to a teleoperator 1808 via a teleoperator computing device 1804. The autonomous vehicle fleet manager 1803 is configured to receive policy data 1802 and environmental data 1806, as well as other data. Further to diagram 1800, the autonomous vehicle communication link manager 1820 is shown to include an environment event detector 1831, a policy conformance determiner 1832, and a movement request processor 1834. The environmental event detector 1831 is configured to receive environmental data 1806 specifying changes in the environment in which the autonomous vehicle service is being implemented. For example, environmental data 1806 may specify that region 1880 has degraded communication service, which may impact autonomous vehicle service. Policy adaptation determinator 1832 may specify parameters to apply when receiving a travel request during such an event (e.g., during a loss of communication). Travel request processor 1834 is configured to process the travel request taking into account the degraded communication. In this example, user 1888 has requested autonomous vehicle service. Furthermore, travel request processor 1834 includes logic for applying an adapted policy to modify how autonomous vehicles are dispatched to avoid complications due to unsatisfactory communication.

[0061] Communications event detector 1840 includes a policy download manager 1842 and a communications-configured ("COMM-configured") AV dispatcher 1844. Policy download manager 1842 is configured to provide an updated policy to autonomous vehicle 1830 taking into account region 1880 of reduced communications, whereby the updated policy can specify a route for quickly exiting region 1880 if the autonomous vehicle enters the region. For example, autonomous vehicle 1864 can receive the updated policy at a time before entering region 1880. Upon loss of communication, autonomous vehicle 1864 implements the updated policy and selects route 1866 for quickly exiting region 1880. COMM-configured AV dispatcher 1844 can be configured to identify points 1865 for parking the autonomous vehicle, configured as relays for establishing a peer-to-peer network across region 1880. As such, the COMM-configured AV dispatcher 1844 is configured to dispatch an autonomous vehicle 1862 (without a passenger) to park at location 1865 for the purpose of operating as a radio tower in a peer-to-peer ad hoc network.

[0062] FIG. 19 is an example flow diagram for determining actions for autonomous vehicles during an event, such as degraded or lost communication, according to some embodiments. Flow 1900 starts at 1901. Policy data is received, whereby the policy data defines parameters to apply to movement requests in a geographic region during the event. At 1902, one or more of the following actions may be performed: (1) dispatch a subset of autonomous vehicles to geographic locations within the portion of the geographic location; the subset of autonomous vehicles park at specific geographic locations, each serving as a static communication relay, or move within the geographic region, each serving as a mobile communication relay; (2) implement peer-to-peer communication between the portion of the pool of autonomous vehicles associated with the portion of the geographic region; (3) provide the autonomous vehicles with an event policy that describes a route for escaping the portion of the geographic region during the event; (4) invoke teleoperation; and (5) recalculate a route to avoid the geographic region. After performing the actions, at 1914, the fleet of autonomous vehicles is monitored.

[0063] FIG. 20 is a diagram illustrating an example of a localizer, according to some embodiments. Diagram 2000 includes a localizer 2068 configured to receive sensor data from sensors 2070, such as lidar data 2072, camera data 2074, radar data 2076, and other data 2078. Additionally, localizer 2068 is configured to receive reference data 2020, such as 2D map data 2022, 3D map data 2024, and 3D regional map data. According to some examples, other map data, such as 4D map data 2025 and semantic map data (not shown), including corresponding data structures and repositories, may also be implemented. Further to diagram 2000, localizer 2068 includes a positioning system 2010 and a localization system 2012, both configured to receive sensor data and reference data 2020 from sensors 2070. The localization data integrator 2014 is configured to receive data from the positioning system 2010 and data from the localization system 2012, whereby the localization data integrator 2014 is configured to integrate or fuse sensor data from multiple sensors to form local location data 2052.

[0064] 21 is an example flow diagram for generating local location data based on aggregated sensor data, according to some embodiments. Flow 2100 starts at 2101. At 2102, reference data is received, where the reference data includes three-dimensional map data. In some examples, reference data such as 3D or 4D map data may be received from one or more networks. At 2104, localization data from one or more localization sensors is received and disposed in a localization system. At 2106, positioning data from one or more positioning sensors is received and disposed in a positioning system. At 2108, the localization data and the positioning data are integrated. At 2110, the localization data and the positioning data are integrated to form local location data that specifies the geographic location of the autonomous vehicle.

[0065] Figure 22 is a diagram illustrating another example of a localizer, according to some embodiments. Diagram 2200 includes a localizer 2268, which includes a positioning system 2210 and a relative positioning system 2212 for generating positioning-based data 2250 and local location-based data 2251, respectively. The positioning system 2210 includes a projection processor 2254a for processing GPS data 2273, GPS datum 2211, and 3D map data 2222, among other optional data (e.g., 4D map data). The positioning system 2210 also includes an odometry processor 2254b for processing wheel data 2275 (e.g., wheel speed), vehicle model data 2213, and 3D map data 2222, among other optional data. Furthermore, position location system 2210 includes integrator processor 2254c for processing IMU data 2257, vehicle model data 2215, and 3D map data 2222, among other optional data. Similarly, relative position location system 2212 includes lidar position location processor 2254d for processing lidar data 2272, 2D tile map data 2220, 3D map data 2222, and 3D local map data 2223, among other optional data. Relative position location system 2212 also includes visual registration processor 2254e for processing camera data 2274, 3D map data 2222, and 3D local map data 2223, among other optional data. Furthermore, relative position location system 2212 includes radar return processor 2254f for processing radar data 2276, 3D map data 2222, and 3D local map data 2223, among other optional data. It should be noted that in various examples, other types of sensor data and sensors or processors may be implemented, such as sonar data, etc.

[0066] Further with respect to diagram 2200, localization-based data 2250 and relative localization-based data 2251 may be provided to data integrator 2266a and localization data integrator 2266, respectively. Data integrator 2266a and localization data integrator 2266 may be configured to fuse corresponding data, whereby localization-based data 2250 may be fused in data integrator 2266a before being fused with relative localization-based data 2251 in localization data integrator 2266. According to some embodiments, data integrator 2266a may be formed as part of localization data integrator 2266 or may be absent. Regardless, both localization-based data 2250 and relative localization-based data 2251 may be provided to localization data integrator 2266 for purposes of fusing the data to generate local position data 2252. Localization-based data 2250 can include unary constraint data (and uncertainty values) from projection processor 2254a and binomial constraint data (and uncertainty values) from odometry processor 2254b and integrator processor 2254c. Relative localization-based data 2251 can include unary constraint data (and uncertainty values) from localization processor 2254d and visual registration processor 2254e, and optionally radar return processor 2254f. According to some embodiments, localization data integrator 2266 can implement nonlinear smoothing functions such as a Kalman filter (e.g., a gated Kalman filter), a relative bundle adjuster, a pose graph relaxation, a particle filter, a histogram filter, etc.

[0067] FIG. 23 is a diagram illustrating an example of a perception engine, according to some embodiments. Diagram 2300 includes a perception engine 2366, which includes a segmentation processor 2310, an object tracker 2330, and a classifier 2360. Additionally, perception engine 2366 is configured to receive, for example, local position data 2352, lidar data 2372, camera data 2374, and radar data 2376. Note that other sensor data, such as sonar data, may be accessed to provide the functionality of perception engine 2366. Segmentation processor 2310 is configured to extract ground plane data and / or segment portions of images to distinguish objects from each other and from static images (e.g., background). In some instances, 3D blobs can be segmented to distinguish them from each other. In some examples, a blob can refer to a set of features that identify an object within a spatially rendered environment and can be composed of elements (e.g., pixels in camera data, points in laser return data, etc.) that have similar properties, such as intensity and color. In some examples, a blob may also refer to a point cloud (e.g., composed of colored laser return data) or other elements that make up an object. The object tracker 2330 is configured to perform frame-by-frame estimation of the movement of blobs, or other segmented image portions. Furthermore, data association is used to associate a blob at one location in a first frame at time t1 with a blob at a different location in a second frame at time t2. In some examples, the object tracker 2330 is configured to perform real-time probabilistic tracking of 3D objects, such as blobs.The classifier 2360 is configured to identify objects and classify them by classification type (e.g., as a pedestrian, bicyclist, etc.) and by energy / activity (e.g., whether the object is dynamic or static), whereby data representing the classification is described by semantic labels. According to some embodiments, probabilistic estimation of object categories can be performed, such as classifying objects as vehicles, bicyclists, pedestrians, etc., with varying confidence for each object class. The perception engine 2366 is configured to determine perception engine data 2354, which can include static object maps and / or dynamic object maps, whereby, for example, a planner can use this information to enhance path planning. According to various examples, one or more of the segmentation processor 2310, the object tracker 2330, and the classifier 2360 can apply machine learning techniques to generate the perception engine data 2354.

[0068] 24 is an example flow diagram for generating perception engine data, according to some embodiments. Flow diagram 2400 begins at 2402, where data representing a local position of an autonomous vehicle is retrieved. At 2404, localization data from one or more localization sensors is received, and at 2406, features of an environment in which the autonomous vehicle is located are segmented to form a segmented object. At 2408, one or more portions of the segmented object are spatially tracked to form at least one tracked object having motion (e.g., estimated motion). At 2410, the tracked object is classified as at least one of a static object or a dynamic object. In some cases, the static object or the dynamic object can be associated with a classification type. At 2412, data identifying the classified object is generated. For example, the data identifying the classified object can include semantic information.

[0069] Figure 25 is an example of a segmentation processor, according to some embodiments. Diagram 2500 shows a segmentation processor 2510 receiving lidar data from one or more lidars 2572 and camera image data from one or more cameras 2574. Local position data 2552, lidar data, and camera image data are received into a meta spin generator 2521. In some examples, the meta spin generator is configured to divide the image into distinguishable regions (e.g., clusters or groups of point clouds) based on various attributes (e.g., color, intensity, etc.), at least two or more of which can be updated simultaneously or near simultaneously. The meta spin data 2522 is used to perform object segmentation and ground segmentation in a segmentation processor 2523, whereby both the meta spin data 2522 and segmentation-related data from the segmentation processor 2523 are applied to a scanned differencing processor 2513. The scanned difference processor 2513 is configured to predict the motion and / or relative velocity of the segmented image portions, which may be used to identify dynamic objects at 2517. Data indicative of the objects with detected velocities at 2517 is optionally transmitted to a planner to enhance path planning decisions. Additionally, data from the scanned difference processor 2513 may be used to approximate the locations of objects to form a mapping of such objects (and, optionally, identify levels of motion). In some examples, an occupancy grid map 2515 may be generated. Data representing the occupancy grid map 2515 may be transmitted to the planner to further enhance path planning decisions (e.g., by reducing uncertainty).2500, image camera data from one or more cameras 2574 is used to classify blobs in a blob classifier 2520, which also receives blob data 2524 from a segmentation processor 2523. The segmentation processor 2510 may also receive raw radar return data 2512 from one or more radars 2576 for segmentation in a radar segmentation processor 2514, which generates radar-related blob data 2516. Continuing with FIG. 25, the segmentation processor 2510 may also receive and / or generate tracked blob data 2518, which is related to the radar data. The blob data 2516, the tracked blob data 2518, the data from the blob classifier 2520, and the blob data 2524 may be used to track an object or portion thereof. According to some examples, one or more of the following may be optional: scanned difference processor 2513, blob classification 2520, and data from radar 2576.

[0070] 26A is a diagram illustrating an example of an object tracker and classifier, according to various embodiments. The object tracker 2630 of diagram 2600 is configured to receive blob data 2516, tracked blob data 2518, data from blob classifier 2520, blob data 2524, and camera image data from one or more cameras 2676. The image tracker 2633 is configured to receive the camera image data from the one or more cameras 2676 to generate tracked image data, which can be provided to a data association processor 2632. As shown, the data association processor 2632 is configured to receive blob data 2516, tracked blob data 2518, data from blob classifier 2520, blob data 2524, and tracking image data from the image tracker 2633, and is further configured to identify one or more associations between the types of data described above. The data association processor 2632 is configured to track, for example, various blob data, frame by frame, for example, to estimate motion, among other things. Furthermore, data generated by the data association processor 2632 can be used by a track updater 2634 to update one or more tracks, or tracked objects. In some examples, the track updater 2634 can implement a Kalman filter or the like to form updated data of the tracked objects, which can be stored online in a track database (“DB”) 2636. Feedback data can be exchanged between the data association processor 2632 and the track database 2636 via path 2699. In some examples, the image tracker 2633 may be optional or may be omitted. The object tracker 2630 may also use other sensor data, such as, for example, radar or sonar, and any other type of sensor data.

[0071] FIG. 26B is a diagram illustrating another example of an object tracker, according to at least some examples. Diagram 2601 includes an object tracker 2631, which may include structure and / or functionality as similarly named elements described in connection with one or more other figures (e.g., FIG. 26A). As shown, object tracker 2631 includes an optional registration portion 2699 that includes a processor 2696 configured to perform object scan registration and data fusion. Processor 2696 is further configured to store the resulting data in a 3D object database 2698.

[0072] Referring back to FIG. 26A , diagram 2600 also includes a classifier 2660, which may include a track classification engine 2662 for generating static obstacle data 2672 and dynamic obstacle data 2674, both of which may be transmitted to a planner for path planning purposes. In at least one example, track classification engine 2662 is configured to determine whether an obstacle is static or dynamic and another classification type of the object (e.g., whether the object is a vehicle, pedestrian, tree, bicyclist, dog, cat, paper bag, etc.). Static obstacle data 2672 may be formed as part of an obstacle map (e.g., a 2D occupancy map), and dynamic obstacle data 2674 may be formed to include a bounding box having data indicating speed and classification type. Dynamic obstacle data 2674, in at least some instances, includes 2D dynamic obstacle map data.

[0073] FIG. 27 is an example of a front-end processor of a perception engine, according to various examples. Diagram 2700 includes a ground segmentation processor 2723a for performing ground segmentation and an over-segmentation processor 2723b for performing “over-segmentation,” according to various examples. Processors 2723a and 2723b are configured to receive lidar data 2775, which is optionally colored. Over-segmentation processor 2723b generates data 2710 of a first blob type (e.g., relatively small blobs), which is provided to an aggregation classification and segmentation engine 2712 that generates data 2714 of a second blob type. Data 2714 is provided to a data association processor 2732, which is configured to detect whether data 2714 exists in a track database 2736. At 2740, a determination is made whether the data 2714 of the second blob type (e.g., a relatively large blob that may contain one or more smaller blobs) is a new track. If so, the track is initialized at 2742; if not, the tracked object data and its track, stored in the track database 2736, can be extended or updated by a track updater 2742. A track classification engine 2762 is coupled to the track database 2736 to identify tracks and update / modify them, for example, by adding, removing, or modifying track-related data.

[0074] FIG. 28 is a diagram illustrating a simulator configured to simulate an autonomous vehicle in a synthetic environment, according to various embodiments. Diagram 2800 includes simulator 2840 configured to generate simulated environment 2803. As shown, simulator 2840 is configured to use reference data 2822 (e.g., 3D map data and / or other maps, or route data including RNDF data or similar road network data) to generate simulated geometry, such as simulated surfaces 2892a and 2892b, within simulated environment 2803. Simulated surfaces 2892a and 2892b may simulate walls or sides of buildings adjacent to roads. Simulator 2840 can also pre-generate or procedurally generate dynamic object data to simulate dynamic agents in the synthetic environment. An example of a dynamic agent is simulated dynamic object 2801, which represents a simulated cyclist having a constant speed. The simulated dynamic agents may optionally respond to other static and dynamic agents in the simulated environment, including the simulated autonomous vehicle. For example, simulated object 2801 may slow down due to other obstacles in simulated environment 2803 rather than following a pre-set trajectory, thereby creating a more realistic simulation of an actual dynamic environment that exists in the real world.

[0075] Simulator 2840 can be configured to generate a simulated autonomous vehicle controller 2847, which includes a synthetic adaptation of perception engine 2866, localizer 2868, motion controller 2862, and planner 2864, each of which can have the functionality described herein within simulated environment 2803. Simulator 2840 can also generate a simulated interface (“I / F”) 2849 to simulate data exchange with different sensor modalities and different sensor data formats. As such, simulated interface 2849 can simulate a software interface for packetized data, for example, from a simulated lidar sensor 2872. Furthermore, simulator 2840 can also be configured to generate a simulated autonomous vehicle 2830 that implements simulated AV controller 2847. Simulated autonomous vehicle 2830 includes a simulated lidar sensor 2872, a simulated camera or image sensor 2874, and a simulated radar sensor 2876. In the example shown, simulated lidar sensor 2872 can be configured to generate a simulated laser that matches ray trace 2892, thereby generating simulated sensor return 2891. Note that simulator 2840 can simulate the addition of noise or other environmental effects on the sensor data (e.g., added diffusion or reflection that affects simulated sensor return 2891). Furthermore, simulator 2840 can be configured to simulate various sensor imperfections, including sensor malfunction, sensor miscalibration, intermittent data outages, etc.

[0076] Simulator 2840 includes a physics processor 2850 for simulating mechanical, static, dynamic, and kinematic aspects of the autonomous vehicle for use in simulating the behavior of simulated autonomous vehicle 2830. For example, physics processor 2850 includes a contact mechanics module 2851 for simulating contact mechanics, a collision detection module 2852 for simulating interactions between simulated bodies, and a multibody dynamics module 2854 for simulating interactions between simulated mechanical bodies.

[0077] The simulator 2840 also includes a simulator controller 2856 configured to, among other things, control the simulation to adapt the function of any synthetically generated elements of the simulated environment 2803 to determine causality. The simulator 2840 includes a simulator evaluator 2858 for evaluating the performance of the synthetically generated elements of the simulated environment 2803. For example, the simulator evaluator 2858 can analyze simulated vehicle commands 2880 (e.g., simulated steering angle and simulated speed) to determine whether such commands are appropriate responses to simulated activity within the simulated environment 2803. Additionally, the simulator evaluator 2858 can evaluate the interaction of the teleoperator 2808 with the simulated autonomous vehicle 2830 via the teleoperator computing device 2804. Simulator evaluator 2858 can evaluate the effect of updated reference data 2827, including updated map tiles and route data, which can be added to guide the response of simulated autonomous vehicle 2830. Simulator evaluator 2858 can also evaluate the response of simulator AV controller 2847 when policy data 2829 is updated, removed, or added. The above description of simulator 2840 is not intended to be limiting. As such, simulator 2840 is configured to perform a variety of different simulations of an autonomous vehicle against a simulated environment, including both static and dynamic characteristics. For example, simulator 2840 can be used to validate software version changes to ensure reliability. Simulator 2840 can also be used to determine vehicle dynamics characteristics and for calibration purposes. Furthermore, simulator 2840 can be used to explore a space of applicable controls and resulting trajectories to perform self-simulation learning.

[0078] FIG. 29 is an example flow diagram for simulating various aspects of an autonomous vehicle, according to some embodiments. Flow diagram 2900 begins at 2902, where reference data including three-dimensional map data is received into a simulator. At 2904, dynamic object data defining the motion patterns of classified objects can be retrieved. At 2906, a simulated environment is formed based on at least the three-dimensional (“3D”) map data and the dynamic object data. The simulated environment can include one or more simulated surfaces. At 2908, an autonomous vehicle is simulated, including a simulated autonomous vehicle controller forming part of the simulated environment. The autonomous vehicle controller can include a simulated perception engine and a simulated localizer configured to receive sensor data. At 2910, simulated sensor data is generated based on data related to at least one simulated sensor return, and at 2912, simulated vehicle commands are generated to cause movement (e.g., vectored propulsion) by the simulated autonomous vehicle within the synthetic environment. At 2914, the simulated vehicle commands are evaluated to determine whether the simulated autonomous vehicle behaved consistent with expected behavior (e.g., consistent with a policy).

[0079] FIG. 30 is an example flow diagram for generating map data, according to some embodiments. Flow diagram 3000 begins at 3002, where trajectory data is retrieved. The trajectory data may include trajectories captured over a period of time (e.g., as logged trajectories). At least localization data may be retrieved at 3004. The localization data may be captured over a period of time (e.g., as logged localization data). At 3006, a camera or other image sensor may be implemented to generate a subset of the localization data. Thus, the retrieved localization data may include image data. At 3008, the subset of the localization data is aligned to identify a global location (e.g., a global position). At 3010, three-dimensional ("3D") map data is generated based on the global position, and at 3012, the three-dimensional map data is available for implementation by, for example, a manual route data editor (e.g., including a manual road network data editor such as an RNDF editor), an automated route data generator (e.g., including an automated road network generator, including an automated RNDF generator), a fleet of autonomous vehicles, a simulator, a teleoperator computing device, and any other components of an autonomous vehicle service.

[0080] FIG. 31 is a diagram illustrating the architecture of a mapping engine, according to some embodiments. Diagram 3100 includes a 3D mapping engine configured to receive trajectory log data 3140, lidar log data 3172, camera log data 3174, radar log data 3176, and other optional logged sensor data (not shown). Logic 3141 includes a loop-closure detector 3150 configured to, among other things, detect whether the sensor data indicates that a nearby point in space has been previously visited. Logic 3141 also includes a registration controller 3152 for aligning map data, including 3D map data in some cases, with one or more registration points. Additionally, logic 3141 provides data 3142 representing the state of the loop closure for use by a global pose graph generator 3143 configured to generate pose graph data 3145. In some examples, pose graph data 3145 can also be generated based on data from a registration refinement module 3146. Logic 3144 includes a 3D mapper 3154 and a lidar self-calibration unit 3156. Additionally, logic 3144 receives sensor data and pose graph data 3145 to generate 3D map data 3120 (or other map data, such as 4D map data). In some examples, logic 3144 can implement a truncated signed distance function ("TSDF") to fuse the sensor data and / or map data to form an optimal three-dimensional map. Additionally, logic 3144 is configured to include texture and reflectance properties.The 3D map data 3120 can be released for use by a manual route data editor 3160 (e.g., an editor for manipulating route data or other types of route or reference data), an automated route data generator 3162 (e.g., logic configured to generate route data or other types of road network or reference data), a fleet of autonomous vehicles 3164, a simulator 3166, a teleoperator computing device 3168, and any other components of an autonomous vehicle service. The mapping engine 3110 can capture semantic information from manual or automatically generated annotations, and from sonar or instrumented environments (e.g., smart stop lamps).

[0081] FIG. 32 is a diagram illustrating an autonomous vehicle application, according to some embodiments. Diagram 3200 illustrates a mobile computing device 3203 including an autonomous service application 3240 configured to contact an autonomous vehicle service platform 3201 to configure transportation for a user 3202 via an autonomous vehicle 3230. As shown, the autonomous service application 3240 can include a transportation controller 3242, which can be a software application residing on a computing device (e.g., a mobile phone 3203, etc.). The transportation controller 3242 is configured to receive, schedule, select, or perform actions associated with an autonomous vehicle and / or fleet of autonomous vehicles from which the user 3202 can configure transportation from the user's location to a destination. For example, the user 3202 can open the application to request a vehicle 3230. The application can display a map, and the user 3202 can drop a pin to indicate their destination, for example, within a geographically fenced area. Alternatively, the application may display a list of nearby pre-specified pickup locations or provide the user with a text input field for typing in a destination by either address or name.

[0082] Further, for the illustrated example, the autonomous vehicle application 3240 may also include a user identification controller 3246 that can be configured to detect that the user 3202 is in a geographic area near or in the vicinity of the autonomous vehicle 3230 when the vehicle is approaching. In some situations, the user 3202 may not easily perceive or identify the autonomous vehicle 3230 for use by the user 3203 when it is approaching (e.g., due to various other vehicles, including trucks, cars, taxis, and other obstacles that are common in urban environments). In an example, the autonomous vehicle 3230 may establish a wireless communication link 3262 (e.g., via radio frequency (“RF”) signals such as WiFi or Bluetooth®, including BLE) to communicate and / or determine the spatial location of the user 3202 relative to the autonomous vehicle 3230 (e.g., using the relative direction and signal strength of the RF signals). In some instances, the autonomous vehicle 3230 may detect the approximate geographic location of the user 3202 using, for example, GPS data or the like. A GPS receiver (not shown) of mobile computing device 3203 can be configured to provide GPS data to autonomous vehicle service application 3240. Thus, user identification controller 3246 can provide GPS data to autonomous vehicle service platform 3201 via link 3260, and autonomous vehicle service platform 3201 can provide its location to autonomous vehicle 3230 via link 3261. Autonomous vehicle 3230 can then determine the relative distance and / or direction of user 3202 by comparing the user's GPS data with the location derived by the vehicle's GPS.

[0083] The autonomous vehicle 3230 may also include additional logic for identifying the presence of the user 3202, such as logic configured to generally detect the user 3202 based on the user's unique facial characteristics or implement a face detection algorithm to identify the user's 3202 identity (e.g., name, phone number, etc.). Additionally, the autonomous vehicle 3230 may include logic for detecting a code for identifying the user 3202. Examples of such codes include specialized visual codes such as QR codes, color codes, etc., specialized audio codes such as voice-activated or recognized codes, etc. In some cases, the code may be an encoded security key that can be digitally transmitted to the autonomous vehicle 3230 via link 3262 to ensure secure entry and / or egress. Additionally, one or more of the above-identified techniques for identifying the user 3202 can be used as a secured means for granting entry and egress privileges to the user 3202 so as to prevent others from entering the autonomous vehicle 3230 (e.g., to ensure that third parties do not enter an unoccupied autonomous vehicle before reaching the user 3202). According to various examples, any other means for identifying user 3202 and providing secured entry and exit may also be implemented in one or more of autonomous vehicle service application 3240, autonomous vehicle service platform 3201, and autonomous vehicle 3230.

[0084] To assist user 3302 in reaching their requested transportation, autonomous vehicle 3230 can be configured to notify or otherwise alert user 3202 of its presence as autonomous vehicle 3230 approaches user 3202. For example, autonomous vehicle 3230 can activate one or more lighting devices 3280 (e.g., LEDs) according to a particular light pattern. In particular, the particular light pattern is generated so that user 3202 can easily perceive that autonomous vehicle 3230 is reserved to service user 3202's transportation request. By way of example, autonomous vehicle 3230 can generate a light pattern 3290 that can be perceived by user 3202 as a "blink" or other animation of its exterior and interior lights in such a visual and temporal manner. Light pattern 3290 may be generated with or without an accompanying audio pattern to identify to user 3202 that this vehicle is the one they have reserved.

[0085] According to some embodiments, the autonomous vehicle user controller 3244 can execute software applications configured to control various functions of the autonomous vehicle. Additionally, the applications can be configured to turn or reroute the autonomous vehicle while traveling to its initial destination. Additionally, the autonomous vehicle user controller 3244 can be configured to cause built-in logic to modify the interior lighting of the autonomous vehicle 3230, for example, to provide mood lighting. The controller 3244 can also control the source of audio (e.g., an external source such as Spotify or audio stored locally on the mobile computing device 3203), select the type of ride (e.g., modify the desired acceleration and braking aggressiveness, modify active suspension parameters to select a set of “road-handling” characteristics to implement aggressive driving characteristics including vibrations, or select a “soft-ride” in which vibrations are damped for comfort), and so on. For example, the mobile computing device 3203 can also be configured to control HVAC functions, such as ventilation and temperature.

[0086] 33-35 illustrate various example computing platforms configured to provide various functions to components of an autonomous vehicle service, according to various embodiments. In some examples, computing platform 3300 can be used to execute computer programs, applications, methods, processes, algorithms, or other software for implementing the techniques described above.

[0087] It should be noted that various structures and / or functions of FIG. 33 are applicable to FIGS. 34 and 35, and therefore some elements of those figures may be discussed in the context of FIG.

[0088] In some cases, the computing platform 3300 may be located within any device, such as a computing device 3390a, an autonomous vehicle 3391, and / or a mobile computing device 3390b, which may be located within one or more computing devices within an autonomous vehicle service platform.

[0089] Computing platform 3300 includes a bus 3302 or other communication mechanism for communicating information, which interconnects subsystems and devices such as processor 3304, system memory 3306 (e.g., RAM, etc.), storage device 3308 (e.g., ROM, etc.), in-memory cache (which may be embodied within RAM 3306 or other portions of computing platform 3300), communication interface 3313 (e.g., Ethernet or wireless controller, Bluetooth controller, NFC logic, etc.) to facilitate communication via ports over communication link 3321 for communicating with computing devices, including, for example, mobile computing devices and / or communication devices having processors. Processor 3304 can be implemented by one or more graphics processing units (“GPUs”), one or more central processing units (“CPUs”) such as those manufactured by Intel® Corporation, or one or more virtual processors, and any combination of CPUs and virtual processors. The computing platform 3300 exchanges data representing input and output via input / output devices 3301, including, but not limited to, a keyboard, a mouse, audio input (e.g., speech-to-text devices), a user interface, a display, a monitor, a cursor, a touch-sensitive display, an LCD or LED display, and other I / O-related devices.

[0090] According to some examples, the computing platform 3300 performs certain operations by the processor 3304 executing one or more sequences of one or more instructions stored in the system memory 3306; the computing platform 3300 may be implemented in a client-server configuration, a peer-to-peer configuration, or as any mobile computing device, including a smartphone or the like. Such instructions or data may be read into the system memory 3306 from another computer-readable medium, such as the storage device 3308. In some examples, hardwired circuitry may be used in place of or in combination with software instructions. The instructions may be embodied in software or firmware. The term "computer-readable medium" refers to any tangible medium that participates in providing instructions to the processor 3304 for execution. Such media may take many forms, including, but not limited to, non-volatile and volatile media. Non-volatile media include, for example, optical or magnetic disks. Volatile media includes dynamic memory, such as the system memory 3306.

[0091] Common forms of computer-readable media include, for example, floppy disks, flexible disks, hard disks, magnetic tape or any other magnetic media, CD-ROMs, any other optical media, punch cards, paper tape, any other physical media with a pattern of holes, RAM, PROM, EPROM, FLASH-EPROM, any other memory chip or cartridge, or any other medium from which a computer can read. Instructions may also be sent or received using transmission media. The term "transmission media" may include any tangible or intangible medium capable of storing, encoding, or carrying instructions for execution by a machine, including digital or analog communications signals or other intangible media for facilitating communication of such instructions. Transmission media include coaxial cables, copper wire, and fiber optics, including the wires that comprise bus 3302 for transmitting computer data signals.

[0092] In some examples, execution of the sequences of instructions can be performed by the computing platform 3300. According to some examples, the computing platform 3300 can be coupled to any other processor by a communications link 3321 (e.g., a wired network such as a LAN, a PSTN, or any wireless network including WiFi, Bluetooth, NFC, Zig-Bee, etc., of various standards and protocols) to perform the sequences of instructions cooperatively with each other (or asynchronously with respect to each other). The computing platform 3300 can send and receive messages, data, and instructions, including program code (e.g., application code), through the communications link 3321 and the communications interface 3313. The received program code may be executed by the processor 3304 as it is received, and / or stored in memory 3306 or other non-volatile storage for later execution.

[0093] In the illustrated example, system memory 3306 can include various modules containing executable instructions for performing the functions described herein. System memory 3306 can include an operating system (“O / S”) 3332, as well as applications 3336 and / or logic modules 3359. In the example shown in FIG. 33, system memory 3306 includes an autonomous vehicle (“AV”) controller module 3350 and / or its components (e.g., a perception engine module, a localization module, a planner module, and / or a motion controller module), any of which, or one or more portions thereof, can be configured to facilitate autonomous vehicle services by performing one or more functions described herein.

[0094] Referring to the example shown in FIG. 34, the system memory 3306 includes an autonomous vehicle service platform module 3450 and / or its components (e.g., teleoperator manager, simulator, etc.), any of which, or one or more portions thereof, can be configured to facilitate management of autonomous vehicle services by performing one or more functions described herein.

[0095] 35, the system memory 3306 includes an autonomous vehicle (“AV”) module and / or components thereof, e.g., for use in a mobile computing device. One or more portions of the module 3550 can be configured to facilitate delivery of autonomous vehicle services by performing one or more functions described herein.

[0096] Referring back to FIG. 33 , the structure and / or functionality of any of the features described above can be implemented in software, hardware, firmware, circuitry, or a combination thereof. It should be noted that the structures and components described above and their functionality may be aggregated with one or more other structures or elements. Alternatively, the elements and their functionality, if any, may be divided into constituent parts. As software, the techniques described above may be implemented using various types of programming or formatting languages, frameworks, syntaxes, applications, protocols, objects, or techniques. As hardware and / or firmware, the techniques described above may be implemented using various types of programming or integrated circuit design languages, including hardware description languages ​​such as any register transfer language ("RTL") configured to design field programmable gate arrays ("FPGAs"), application specific integrated circuits ("ASICs"), or any other type of integrated circuit. According to some embodiments, the term "module" can refer, for example, to an algorithm or portion thereof, and / or logic implemented in either hardware circuitry or software, or a combination thereof. These may be varied and are not limited to the examples or descriptions provided.

[0097] In some embodiments, module 3350 of FIG. 33, module 3450 of FIG. 34, and module 3550 of FIG. 35, or one or more of their components, or any process or device described herein, may be in communication (e.g., wired or wireless) with or may be located within a mobile device, such as a mobile phone or mobile computing device.

[0098] In some cases, a mobile device or any network-connected computing device (not shown) (or any process or device described herein) that communicates with one or more modules 3359 (module 3350 of FIG. 33 , module 3450 of FIG. 34 , and module 3550 of FIG. 35 ) or one or more of its components can provide at least some of the structure and / or functionality of any of the features described herein. As shown in the figures described above, the structure and / or functionality of any of the features described above can be implemented in software, hardware, firmware, circuitry, or any combination thereof. Note that the above structures and components and their functionality may be aggregated or combined with one or more other structures or elements. Alternatively, the elements and their functionality, if any, can be divided into constituent parts. As software, at least some of the techniques described above may be implemented using various types of programming or formatting languages, frameworks, syntaxes, applications, protocols, objects, or techniques. For example, at least one of the elements shown in any of the figures can represent one or more algorithms. Alternatively, at least one of the elements may represent portions of logic, including portions of hardware configured to provide the configuration structure and / or functionality.

[0099] For example, module 3350 of FIG. 33, module 3450 of FIG. 34, and module 3550 of FIG. 35, or one or more of their components, or any process or device described herein, can be implemented in one or more computing devices (i.e., any mobile computing device, whether worn or carried, such as a wearable device, an audio device (such as headphones or a headset), or a mobile phone) including one or more processors configured to execute one or more algorithms in memory. Thus, at least some of the elements in the figures described above can represent one or more algorithms. Or, at least one of the elements can represent portions of logic, including portions of hardware configured to provide a configuration structure and / or function. These may vary and are not limited to the examples or descriptions provided.

[0100] As hardware and / or firmware, the construction techniques described above can be implemented using various types of programming languages ​​or integrated circuit design languages, including hardware description languages ​​such as any register transfer language ("RTL") configured to design a field programmable gate array ("FPGA"), an application specific integrated circuit ("ASIC"), a multi-chip module, or any other type of integrated circuit.

[0101] For example, module 3350 of Figure 33, module 3450 of Figure 34, and module 3550 of Figure 35, or one or more of their components, or any process or device described herein, may be implemented in one or more computing devices including one or more circuits. Thus, at least one of the elements in the above-described figures may represent one or more components of hardware. Alternatively, at least one of the elements may represent portions of logic, including portions of circuitry configured to provide a configuration structure and / or functionality.

[0102] According to some embodiments, the term "circuit" can refer to any system including several components, e.g., through which current flows to perform one or more functions, where components include discrete and complex components. Examples of discrete components include transistors, resistors, capacitors, inductors, diodes, etc., and examples of complex components include memory, processors, analog circuits, digital circuits, etc., including field programmable gate arrays ("FPGAs") and application specific integrated circuits ("ASICs"). Thus, a circuit can include a system of electronic and logical components (e.g., logic configured to execute instructions, e.g., a group of executable instructions of an algorithm, and thus a component of a circuit). According to some embodiments, the term "module" can refer to, e.g., an algorithm or portions thereof, and / or logic implemented in either hardware circuitry or software, or a combination thereof (i.e., a module can be implemented as a circuit). In some embodiments, an algorithm and / or a memory in which an algorithm is stored is a "component" of a circuit. Thus, the term "circuit" can also refer to, e.g., a system of components including an algorithm, which can be varied and is not limited to the examples or descriptions provided.

[0103] FIG. 36 is a diagram illustrating a mapping engine configured to adaptively generate mapping data for an autonomous vehicle in response to changes in the physical environment, according to some examples. Diagram 3600 shows a mapping engine 3654 disposed within an autonomous vehicle service platform 3601 communicatively coupled to one or more autonomous vehicles 3630 via a communications layer (not shown). Mapping engine 3654 is configured to generate map data and adaptively modify the map data in response to changes in the physical environment traversed by the autonomous vehicles 3630. In the illustrated example, mapping engine 3654 can generate mapping data based on sensor data received from autonomous vehicles 3630, which are shown as having any number of sensors or sensor devices 3604a, 3604b, and 3604c of sensor type 3602a, sensor type 3602b, and sensor type 3602c, respectively. Autonomous vehicles 3630 may include any number of other sensors or sensor devices 3604n having any other sensor type 3602n. Sensors 3604a, 3604b, 3604c, and 3604n generate sensor data 3607a, 3607b, 3607c, and 3607n, respectively, one or more of which may be received by mapping engine 3654 for generating map data 3659 (e.g., 2D, 3D, and / or 4D map data). Map data 3659 may be transmitted to autonomous vehicle 3630 for storage in map repository 3605a and for use to facilitate localization and other functionality. Specifically, autonomous vehicle 3630 may include a localizer (not shown) that uses the map data in map repository 3605a to determine the position and / or local location of the autonomous vehicle at any time, including while traveling.

[0104] In view of the foregoing, the structure and / or functionality of the mapping engine 3654 and its components can facilitate the generation of “self-healing” maps and map data, for example, by detecting changes in portions of the map data over time and generating updated maps (i.e., updated map data) that include variations or changes to the physical environment traversed by the autonomous vehicles 3630. In some implementations, the mapping engine 3654 can generate an adaptive three-dimensional model of the cityscape adjacent to the network of routes and roads traveled by the fleet of autonomous vehicles. The 3D model of the portion of the cityscape can be derived by identifying data representing surfaces (and other surface attributes, such as surface shape, size, texture, color, etc.) that make up the facade or exterior of the objects, such as buildings (including commercial signs), trees, guardrails, barriers, streetlights, traffic signs and signals, and any other physical features that can be detected by the sensors 3604a, 3604b, 3605c, and 3604n. Accordingly, mapping engine 3654 may be configured to detect objects (or absence of objects) associated with a portion of the map data as well as changes in the objects (e.g., changes in color, size, etc.), and may be further configured to incorporate changes to the objects into the map data to adaptively (e.g., automatically) form an updated portion of the map data. Accordingly, the updated portion of the map data may be stored in map repository 3605a to, among other things, improve the accuracy of positioning functions (as well as other autonomous vehicle controller functions, including planning, etc.) for autonomous vehicle 3630.

[0105] In some cases, the map data 3659 generated by the mapping engine 3654 may be used in combination with locally generated map data (not shown), such as generated by a local map generator (not shown) in the autonomous vehicle 3630. For example, the autonomous vehicle controller (not shown) may detect that one or more portions of the map data in the map repository 3605a change from one or more portions of the locally generated map data. Logic in the autonomous vehicle controller may analyze the differences in the map data (e.g., variation data) to identify changes in the physical environment (e.g., additions, removals, or changes in static objects). In some examples, the term “variation data” may refer to differences between the remotely generated map data and the locally generated map data. Based on the changed portions of the environment, the autonomous vehicle controller may implement different proportional amounts of the map data in the map repository 3605a and the locally generated map data to optimize localization. For example, the autonomous vehicle controller may generate blended map data composed of both remotely generated and locally generated map data to optimize the determination of the position or local location of the autonomous vehicle 3630. Additionally, upon detecting variable data, the autonomous vehicle controller may cause transmission (at different bandwidths or data rates) of varying amounts of sensor-based or other data to the autonomous vehicle service platform 3601. For example, the autonomous vehicle service platform 3601 may receive different types of data at different data rates based, for example, on the importance of receiving guidance from a teleoperator.As another example, a subset of the sensor data 3607a, 3607b, 3607c, and 3607n may be transmitted (e.g., at an appropriate data rate) to, for example, modify the map data to form various degrees of updated map data in real time (or near real time), and to further perform one or more of the following: (1) evaluate and characterize differences in the map data, (2) propagate updated portions of the map data to other autonomous vehicles in the fleet, (3) generate a notification to the teleoperator computing device in response to detecting differences in the map data, and (4) generate a representation of the environment (and changed portions thereof) as sensed by the various sensor devices 3604a, 3604b, 3604c, and 3604n for display at any sufficient resolution in the user interface of the teleoperator computing device. It should be noted that the examples described above are not limiting, and any other map-related functionality for managing a fleet of autonomous vehicles may be implemented using the mapping engine 3654 taking into account detected changes in the physical environment compared to the map data.

[0106] According to some examples, sensor type 3602a, sensor type 3602b, and sensor type 3602c can include a laser-based sensor, an image-based sensor, and a radar-based sensor, respectively. As such, sensors 3604a, 3604b, and 3604c can include a lidar, a camera, and a radar device, respectively. As shown in diagram 3600, multiple sensor devices (e.g., lidars) 3604a each generate different laser-based sensed data 3607a at a different geographic location. For example, each lidar 3604a can be positioned at a different location on the autonomous vehicle 3630, and each can be oriented differently (see FIGS. 3A and 3C, both of which show different lidars with different views and sensor fields of view). Given the directional nature of the projecting laser beams, different laser returns from different lidars 3604a may return from a common point (or a common set of points associated with a traffic sign, for example) at different times. Mapping engine 3654 and / or components of autonomous vehicle service platform 3601 may be configured to align, map, transform, or correlate the laser returns of different lidars 3604a to common points of laser returns from surfaces in the environment. Mapping engine 3654 and / or components of autonomous vehicle service platform 3601 may also process sensor data 3607b and sensor data 3607c in a similar manner.

[0107] In some examples, the one or more sensors 3604n can include a variety of different sensor types (“n”) 3602n to generate a variety of different subsets of sensor data 3607n. Examples of sensors 3604n include positioning sensors, such as one or more Global Positioning System (“GPS”) data receiver-sensors, one or more Inertial Measurement Units (“IMUs”), one or more odometry sensors (e.g., wheel encoder sensors, wheel speed sensors, and the like), one or more wheel angle sensors, and the like that provide position and attitude data for the autonomous vehicle. Such attitude data can include one or more coordinates (e.g., x-coordinate, y-coordinate, and / or z-coordinate), yaw value, roll value, pitch value (e.g., angle value), rate (e.g., speed), altitude, and the like.

[0108] Log data repository 3609 within autonomous vehicle service platform 3601 is configured to receive and store subsets of sensor data 3607a, 3607b, 3607c, and 3607n, which, in at least one example, comprise raw lidar data, raw camera data, raw radar data, and other raw sensor data, respectively. As shown in diagram 3600, at a common point in time or during a common time period, the subsets of sensor data 3607a, 3607b, and 3607c may be stored or recorded as dataset ("1") 3610a, dataset ("2") 3610b, and dataset ("n") 3610n, or as any number of datasets. Datasets 3610a, 3610b, and 3610n may be stored within a log file data structure, according to some examples. Additionally, sensor data 3607n, which may be sensed simultaneously with a subset of sensor data 3607a, 3607b, and 3607c, may also be stored as part of the log files for data sets 3610a, 3610b, and 3610n.

[0109] Alignment controller 3640 may be configured to receive one or more of sensor data 3607a, 3607b, 3607c, and 3607n, as well as other data 3603m. Alignment controller 3640 may also be configured to generate data representing an aligned subset of sensor data 3607a, 3607b, 3607c, and 3607n. In some cases, sensor data 3607 may include a subset of sensor data 3607n that includes positioning data (e.g., sensor data 3607m may include GPS, IMU, and odometry data). With respect to sensor data alignment, examples of data representing an aligned subset of sensor data include data representing at least aligned lidar data and aligned camera data. According to some examples, alignment controller 3640 may be configured to implement an alignment algorithm to align the sensor data by aligning portions or frames of lidar sensor data and identifying “alignment” points that align portions or frames of camera data. For example, the alignment controller 3640 can map or associate laser returns from one lidar to another, and pixel data from one camera to another. Further, the alignment controller 3640 can generate positioning map data, and such data can be stored in a data structure based on a pose graph model in which data specifying individual poses (e.g., local positions) can be spatially correlated based on positioning sensor data (e.g., GPS data, IMU data, odometry data, etc.) collected from sensors 3607 n.

[0110] The mapping engine 3654 may be configured to receive the aligned sensor data (e.g., registered sensor data) and positioning map data (e.g., data related to a pose graph) described above to generate a high-definition ("HD") three-dimensional model of the cityscape adjacent to the network of roads based on integration of a subset of the sensor data 3607a, 3607b, 3607c, and 3607n. As shown in diagram 3600, the mapping engine 3654 may include one or more of the following, according to various examples: an integrator 3651 for integrating the sensor data; a calibrator 3652 for calibrating the sensor data; a data change detector 3653 for detecting changes in portions of the map data; a tile generator 3656 for generating formatted map data; and a data change manager 3657 for managing the implementation of the changed map data.

[0111] Integrator 3651 may be configured to integrate multiple subsets of sensor data (e.g., of the same or different sensor modalities) to generate high-resolution (e.g., relatively high-resolution) image data as a 3D model of the environment in which the autonomous vehicle travels, and may further be configured to reduce errors associated with individual types of sensors. According to some examples, integrator 3651 is configured to fuse sensor data (e.g., lidar data, camera data, radar data, etc.) to form integrated sensor data. Furthermore, raw sensor datasets 3610a, 3610b, and 3610n may be received from one or more autonomous vehicles 3630 to fuse a collection of one or more subsets of sensor data of one or more sensor modalities from a fleet of autonomous vehicles 3630. By fusing the data from raw sensor datasets 3610a, 3610b, and 3610n, integrator 3651 can generate a 3D dataset including the fused sensor data, such as dataset("1") 3655a and dataset("2") 3655b. The integrator 3651 can integrate or otherwise fuse at least two types of sensor data, including a subset of laser return data and a subset of image data. In some examples, fusing the laser and image data can include correlating pixel data of the subset of image data with the subset of laser return data. Optionally, the integrator 3651 can associate pixel data of one or more pixels with one or more laser returns, thereby associating the laser data with a portion of a surface within the three-dimensional tile data. The pixel data can specify one or more surface characteristics, including texture, color, reflectivity, transparency, etc. According to some examples, the integrator 3651 can perform a Kalman filtering process or a variant thereof (e.g., an extended Kalman filtering process), or any other process for fusing sensor data.Integrator 3651 may also include logic for extracting or otherwise determining surface characteristics related to surfaces or features of objects (e.g., buildings, trees, parked cars, etc.), as well as the pose of the autonomous vehicle from which sensor data may be acquired.

[0112] Integrator 3651 may be configured to use sensor datasets 3655a and 3655b to extract data related to the surfaces of physical objects in the autonomous vehicle's environment. Datasets 3655a and 3655b, as well as others not shown, may include fused sensor data representing three-dimensional models related to different points in time or different time intervals. Thus, datasets 3655a and 3655b may be used to detect whether there are changes in the physical environment or portions thereof over time. Note that in at least some implementations, integrator 3651 may also perform a distance transform, such as a signed distance function (“SDF”), to determine one or more surfaces external to the autonomous vehicle. In one example, a truncated signed distance function (“TSDF”) or equivalent may be performed to identify one or more points on the surface relative to a reference point (e.g., one or more distances to a point on the surface of an external object relative to a local location).

[0113] The integrator 3651 may be configured to generate a 3D model of the cityscape (or any external object features) as a probability map, whereby the map data can represent a probability distribution over one or more environmental characteristics. For example, the probability map may be formed using laser intensity (e.g., average laser intensity or reflectance) and variations in infrared remittance values ​​at distances or points in space related to the pose of the autonomous vehicle. A data structure for storing map data may include several cells containing, for example, intensity mean values ​​and variance values. In some examples, this data structure or any other data structure may also include several cells for storing 3D map data such as color data (e.g., RGB values ​​or other color space values), texture data, reflectance data, or any other surface characteristic or attribute data (e.g., specularity data). Cells configured to store data related to the map may be implemented as voxels or as 3D tiles, according to some examples.

[0114] The mapping engine 3654 and / or integrator 3651, as well as other components of the mapping engine 3654, may be configured to generate 3D maps in an “offline” mode of operation. For example, the mapping engine 3654 may implement algorithms (e.g., machine learning, including deep learning algorithms) that analyze a dataset 3655 based on a recorded dataset (e.g., static data) to generate map data. However, the mapping engine 3654 may not be limited to offline map generation and may also implement “online” map generation techniques in which one or more portions of raw sensor data may be received in real time (or near real time) to generate or identify changes to map data. The mapping engine 3654 may implement logic configured to perform simultaneous localization and mapping (“SLAM”) or any suitable mapping technique.

[0115] Data change detector 3653 is configured to detect changes in datasets 3655a and 3655b, which are examples of any number of datasets of 3D map data. Data change detector 3653 is also configured to identify portions of the map data that have changed and, optionally, generate data that identifies or classifies objects associated with the changed portions of the map data. In the shown example, some datasets, including dataset 3655a, include map data configured to generate map data conceptually shown as 3D model data 3660 (e.g., roads at time T1 that include portions of map data 3664). However, at time T2, data change detector 3653 can detect that another number of datasets, including dataset 3655b, include data representing the presence of an external object within portions of map data 3665 of 3D model data 3661, thereby causing portions of map data 3665 to match portions of map data 3664 at a different time. Thus, the data change detector 3653 can detect changes in the map data and can further adaptively modify the map data to include the changed map data (e.g., as updated map data).

[0116] According to some examples, data change detector 3653 is configured to execute one or more statistical change detection algorithms to detect changes in the physical environment. Multiple temporal analysis techniques or other suitable algorithms may also be used. The structure of datasets 3655a and 3655b may be implemented as a cumulative data structure for indexing sensor data (e.g., measurements thereof) stored within the 3D map data structure. By way of example, the statistical change detection algorithm may be configured to detect portions of the map data that have changed by identifying boundaries over one or more iterations of a deep learning calculation. Specifically, data change detector 3653 may be configured to detect boundaries of map data portions 3664 and 3665 over time, such as across two or more datasets (e.g., over one or more passes or epochs of application of the datasets to the statistical change detection algorithm or deep learning algorithm). Epoch determination may also be applied, for example, to construct a 4D map and associated 4D map data. In some examples, data change detector 3653 may classify portions of the map data as well as objects therein to identify whether the objects are static or dynamic. In some cases, dynamic objects may be excluded from map data generation.

[0117] The mapping engine 3654 is configured to provide map data 3659 to the map data repository 3605a within the reference data repository 3605. The mapping engine 3654 may be configured to apply changes in the map data to form updated three-dimensional (“3D”) map data as reference data for transmission to a reference data store (i.e., repository) within the fleet of autonomous vehicles. The changes in the data may represent state changes in the environment in which various types of sensor data are sensed. Accordingly, state changes in the environment may indicate changes in the state of objects located therein (e.g., the inclusion of data representing the presence or absence of one or more objects). In some examples, the data change manager 3657 may be configured to identify or otherwise designate (e.g., via identifier or indicator data 3658) that a portion of the map data includes modified map data 3658 (or a representation thereof). As shown, map data 3692 stored in the map repository 3605a is associated or linked to representation data (“delta data”) 3694 indicating that the relevant portion of the map data has been modified. Further to the example shown, display data 3694 may identify a set of traffic cones placed within the physical environment associated with 3D model 3661 in which the autonomous vehicle travels as a modified portion of map data 3665.

[0118] Tile generator 3656 may be configured to generate two-dimensional or three-dimensional map tiles based on the map data from datasets 3655a and 3655b. The map tiles may be transmitted for storage in map repository 3605a. Tile generator 3656 may generate map tiles including indicator data to indicate that a portion of the map is an updated portion of the map data. Furthermore, the updated map portion may be incorporated into reference data repository 3605 within the autonomous vehicle. Thus, consider an example in which autonomous vehicle 3630 travels through a physical environment and plans to travel near a recently added object (e.g., a traffic cone) in the environment. A localizer (not shown) can access map data associated with the changed portion of the map data (e.g., the updated portion of the map data) to localize the autonomous vehicle. Upon detecting a localization performed with an updated map version, logic can perform additional processing to ensure that use of the updated map data can be used to navigate autonomous vehicle 3630 efficiently and safely. For example, when a map tile containing modified map data is accessed or performed during location determination, a request for monitoring or assistance may be generated for a teleoperator. Note that in some instances, modified portions of map data may also refer to temporary map data, as such data may be used in fewer situations than, for example, verified map data.

[0119] Note, however, that modified portions of map data may also be verified for integration into the map data, thereby transitioning the status of the modified map data from “temporary” to “verified.” To illustrate an example of verifying such data, consider that changes in map data may be exported to a simulator computing device as updated three-dimensional map data. The simulator computing device may then simulate the performance of a portion of a fleet of autonomous vehicles in a simulated environment based on the updated three-dimensional map data. Upon verifying the updated three-dimensional map data, the modified map portions may be incorporated to form new three-dimensional map data. The “new” three-dimensional map data may be viewed as three-dimensional map data that may be trusted, such that the display of the modified map data (i.e., the display of modified map data 3694) may be removed along with the invocation of a request for teleoperator assistance (e.g., an automated request).

[0120] According to some examples, the mapping engine 3654 can include or be implemented as a 3D mapping engine and / or mapper such as that shown in FIG. 31. Furthermore, the components of the mapping engine 3654 may be combined within or without the mapping engine 3654, or may be otherwise distributed. The mapping engine 3654 and any of its components may be implemented in hardware or software, or any combination thereof. Furthermore, the mapping engine 3654 may include any functionality and / or structure described herein, including one or more components of a perception engine for performing object detection, segmentation, and / or classification.

[0121] As a further example, consider that alignment controller 3640 may include one or more components of mapping engine 3110 of FIG. 31. For example, alignment controller 3640 may include loop-closure detector 3150, alignment controller 3152, global pose generator 3143, and alignment refinement module 3146. In the example shown in FIG. 36, autonomous vehicle service platform 3601 may implement loop-closure detector 3150 of FIG. 31 as part of alignment controller 3640, which may be configured to detect one or more portions of a position graph previously traveled by autonomous vehicle 3630 of FIG. 36 (e.g., loop closure detector 3150 of FIG. 31 may perform one or more loop closure processes to identify closed loops). Alignment controller 3152 may be configured to align or register multiple portions or frames of the same or different sensor data. For example, one or more datasets of image data may be transformed or otherwise mapped to one another and to one or more datasets of laser return data and / or radar return data. The alignment controller 3152 may be configured to align subsets of laser return data, subsets of image data, and the like based on trajectory data representing the position data to identify relative coordinates in a global coordinate system. Examples of trajectory data include GPS data, IMU data, odometry data, etc. The global pose graph generator 3143 may be configured to generate position graph data 3145 to specify the position of the autonomous vehicle 3630 of FIG. 36 relative to a global coordinate system. Thus, the locally detected positions of the position graph may be referenced to the global coordinate system. For example, the global position graph generator 3143 of FIG. 31 may be configured to form a global position graph referenced to the global coordinate system.The global position graph may be formed based on the first type of sensor data (e.g., a subset of laser return data) and the second type of sensor data (e.g., a subset of image data), as well as other optional sensor data (e.g., a subset of radar data). Furthermore, the global position graph generator 3143 may also be configured to align the subset of laser return data and the subset of image data to positions relative to coordinates of a global coordinate system. The alignment refinement module 3146 is configured to refine the alignment of one or more of the captured image data, the captured laser return data, or other captured sensor data, such as radar data and the like. In some examples, the alignment refinement module 3146 is configured to reduce or remove artifacts in the map data (e.g., blur artifacts or the like), for example, following projection of color data onto a 3D mapped surface.

[0122] FIG. 37 is a diagram illustrating an example of an autonomous vehicle controller implementing updated map data, according to some examples. Diagram 3700 shows a mapping engine 3754 configured to generate map data 3759, which may be implemented as three-dimensional map tiles. In the example shown, the map data 3759 may also include modified map data 3758, which includes either a portion of the map data that has changed (e.g., an updated portion of the map data for use with an unchangeable portion of the map data) or an indication (e.g., indicator data or a pointer) that identifies the updated portion of the map data that has changed, or both. Further to diagram 3700, autonomous vehicle service platform 3701 may be configured to transmit map data 3786 and modified map data 3788 over network 3702. Autonomous vehicle controller 3747 uses map data 3786 and / or modified map data 3788 to locate autonomous vehicle 3730. In some examples, autonomous vehicle controller 3747 may detect that modified map data 3788 is being accessed during location determination. The autonomous vehicle controller 3747 can then generate teleoperator request data 3770 to request assistance from a teleoperator. The teleoperator request data 3770 may also be configured to request that the teleoperator monitor the performance of at least the autonomous vehicle 3730 during location determination when an updated portion of the map data is accessed or implemented (or when the autonomous vehicle 3730 approaches or travels near a physical location associated with the updated portion of the map data).

[0123] In some examples, the mapping data generated by the mapping engine 3754 may be used to generate other reference data, such as route data (e.g., road network data), mission data, such as MDF-like data, and other reference data that may be used to navigate a fleet of autonomous vehicles, such as RNDF-like data. As shown, a route data generator 3780 may be configured to generate route data 3782 based on unmodified and / or validated map data. Additionally, the route data generator 3780 may be configured to generate modified route data 3784, which may be generated using modified and / or unvalidated map data. In some cases, the autonomous vehicle controller 3747 may generate teleoperator requested data 3770 in response to detecting the use of the modified route data 3784. Thus, the modified route data 3784 (e.g., unvalidated or temporary map data) may be used to navigate an autonomous vehicle with or without the assistance of teleoperator-generated guidance data.

[0124] FIG. 38 is a flow diagram illustrating an example of generating map data, according to some embodiments. Flow 3800 starts at 3802. A subset of multiple types of sensor data is accessed at 3802 (e.g., in a data store or repository that may include log files). The subset of multiple types of sensor data may correspond to a group of multiple sensors or sensor devices. For example, a subset of lidar sensor data may correspond to a group of different lidar sensors from which laser return data is received. At 3804, the sensor data may be aligned with respect to a global coordinate system to form aligned sensor data. For example, a positioning process or algorithm may be configured to align or position the sensor data. At 3806, a dataset of three-dimensional map data may be generated based on the aligned sensor data. At 3808, changes in the map data may be detected for at least two datasets of three-dimensional map data. The changes in the map data may be applied at 3810 to form updated three-dimensional map data. One or more updated portions of the 3D map data may be formatted as reference data for transmission to one or more vehicles in a fleet of autonomous vehicles. At 3812, the updated (e.g., changed) three-dimensional map data may be transmitted to the at least one autonomous vehicle. Note that the order shown in this flow diagram or other flow diagrams herein is not intended to imply a requirement for performing various functions in a linear manner, as portions of the flow diagram may be performed independently or dependently of other portions of the flow diagram, as well as sequentially or concurrently with any one or more other portions of the flow diagram.

[0125] 39 is a diagram illustrating an example of a localizer configured to implement map data and locally generated map data, according to some examples. According to various examples, a localizer 3968 of an autonomous vehicle (“AV”) controller 3947 may be configured to generate local position data 3920 based on either locally generated map data 3941 or map data 3943, or a combination thereof. The local position data 3920 may include data describing a local position of an autonomous vehicle 3930, and the map data 3943 may be generated in a mapping engine 3954 of the autonomous vehicle service platform 3901. Thus, the localizer 3968 can use the map data 3943 to perform localization taking into account changes, deviations, or variations between the locally generated map data 3941 and the map data 3943.

[0126] Diagram 3900 shows an autonomous vehicle 3930 that includes an autonomous vehicle controller 3947, a local map generator 3940, and a reference data repository 3905. Diagram 3900 also shows an autonomous vehicle service platform 3901 that includes a mapping engine 3954 and a teleoperator computing device 3904. The reference data repository 3905 includes a map store 3905a configured to store three-dimensional map data 3943 and a route data store 3905b that may be a data repository for storing route data (e.g., with or without an indication that a portion of the route data or road network data is associated with changed or updated road network data).

[0127] The local map generator 3940 may be configured to receive multiple quantities and types of sensor data, such as sensor data from sensor types 3902a, 3902b, and 3902c. According to various examples, the local map generator 3940 may be configured to locally generate map data (e.g., three-dimensional map data) in real time (or near real time) based on the sensor data from sensor types 3902a, 3902b, and 3902c (e.g., from a group of lidar sensors, a group of cameras, a group of radar, etc.). The local map generator 3940 may implement logic configured to perform simultaneous localization and mapping (“SLAM”) or any suitable mapping technique. In at least some examples, the local map generator 3940 may implement an “online” map generation technique in which one or more portions of raw sensor data from sensor types 3902a-3902c may be received in real time (or near real time) to generate map data (or identity changes thereto) for navigating the autonomous vehicle 3930. The local map generator 3940 may also perform a distance transform, such as a signed distance function ("SDF"), to determine the surface exterior to the autonomous vehicle. In one example, a rounded signed distance function ("TSDF") or equivalent may be performed to identify one or more points on the surface relative to a reference point (e.g., one or more distances to a point on the surface of an external object), whereby the TSDF function may be used to fuse the sensor data and the surface data to form three-dimensional local map data 3941.

[0128] The localizer 3968 may be configured to receive sensor data as well as the locally generated map data 3941 and the map data 3943 to locate the autonomous vehicle 3930 relative to coordinates of a global coordinate system associated with the three-dimensional map data 3943 (or any other reference data). The localizer 3968 is also shown to include a variation detector 3969a and a blending map selection controller 3969b. The variation detector 3969a is configured to compare the locally generated map data 3941 with the map data 3943 to determine whether a portion of the map data associated with a particular surface or point in space changes. Specifically, the variation detector 3969a can detect that data (e.g., variation data) representing two or more map portions of the local map data 3941 varies from the three-dimensional map data 3943.

[0129] Upon detecting a fluctuating map data portion or variation data, the localizer 3968 may be configured to localize the autonomous vehicle 3930 using blended map data from the locally generated map data 3941 and map data 3943. In the example shown, the blended map selection controller 3969b is configured to control whether the locally generated map data 3941 or map data 3943, or a combination thereof, may be used for localization. According to some examples, different amounts of the locally generated map data 3941 and map data 3943 may be used based on corresponding probability distributions that may indicate, for example, the reliability or accuracy of each. In some examples, the blended map selection controller 3969b may be configured to characterize differences between one or more map portions of the map data 3943 and one or more portions of the local map data 3941 to form variation data. Based on the variation data, the blending map selection controller 3969b may be configured to determine a priority of using the local map data 3941 and a priority of using the map data 3943, and may be further configured to cause the localizer 3968 to use a first prioritized amount of the local map data 3941 and a second prioritized amount of the three-dimensional map data 3943 based on the variation data. As an example, consider an example in which the variation detector 3969a detects variation data for some portions of the map data 3943 that vary from corresponding portions of the local map data 3941. Consider further that the local map data 3941 is determined to be more accurate for most portions of the variation data. However, at least one portion of the local map data 3941 has a relatively lower likelihood of being accurate than the corresponding portion of the map data 3943.In this case, the blending map selection controller 3969b may rely more on the local map data 3941 for position determination (with some dependence on the map data 3943), but may also rely more on a particular portion of the map data 3943 (e.g., having a higher priority) for position determination than a corresponding portion of the local map data 3941 (e.g., having a lower priority).

[0130] 40 is a diagram illustrating an example of a localizer configured to vary the transmission rate or amount of locally generated sensor data and / or map data, according to some examples. Diagram 4000 illustrates several autonomous vehicles, including autonomous vehicles 4030a, 4030b, 4030c, and 4030n, and also illustrates an autonomous vehicle service platform 4001 including a mapping engine 4054 and teleoperator logic 4004 implemented in association with a teleoperator computing device 4006 that accepts data signals (e.g., user input) from a teleoperator 4008. The teleoperator logic 4004 may be located on a server computing device (not shown) or within the teleoperator computing device 4006. As shown, the autonomous vehicle 4030a can include an autonomous vehicle controller 4047, a reference data repository 4005 (e.g., including a map store or repository 4005a for storing map data 4046 and a route data store or repository 4005b), and a transceiver 4044 configured to exchange data between the autonomous vehicle 4030a and the autonomous vehicle service platform 4001. Further adding to diagram 4000, the autonomous vehicle controller 4070 can include a local map generator 4040 that can be configured to generate local map data 4041 based on sensor data from different types of sensors 4002a-4002c. The autonomous vehicle controller 4070 is also shown to include a localizer 4068, which is shown to include a fluctuation detector 4069a and a communication controller 4069b, for generating the local position data 4020. It should be noted that the elements shown in diagram 4000 of FIG. 40 may include structure and / or functionality as similarly named elements described in connection with one or more other drawings, such as FIG. 39, among others.

[0131] Following detecting a variation between the local map data 4041 and the map data 4043 (generated by the mapping engine 4054), the communications controller 4069b may be configured to control the transceiver 4044 and the type or amount of data transmitted to the autonomous vehicle service platform 4001. Thus, the communications controller 4069b, according to various examples, is configured to provide sufficient data for the teleoperation logic 4004 and / or the teleoperator 4008 to select an optimal set of guidance data to resolve the detected map data variation. The communications controller 4069b is configured to provide an optimal amount of data or data rate to conserve bandwidth. To illustrate the operation of the communications controller 4069b, consider that the variation detector 4069a detects a relatively minor or small amount of difference between the map data 4043 and the local map data 4041. In this case, the communications controller 4069b may transmit a relatively small amount of data to provide a warning to the teleoperator 4008 to prompt the teleoperator to at least monitor the autonomous vehicle 4030a as it navigates through an environment containing minor changes. Additionally, during a degraded or low-speed data communications connection, simpler or more abstract representations of the data may be transmitted (e.g., bounding boxes with associated metadata) rather than larger amounts of data.

[0132] As another example, consider that fluctuation detector 4069a detects a relatively modest amount of difference between map data 4043 and local map data 4041. In this case, communications controller 4069b can be configured to increase the transmission bandwidth of transceiver 4044 to transmit one or more portions of local map data 4041 to autonomous vehicle service platform 4001 for evaluation by teleoperator logic 4004. In yet another example, consider that fluctuation detector 4069a detects a relatively large amount of difference between map data 4043 and local map data 4041. In this case, communications controller 4069b can be configured to further increase the transmission bandwidth by transceiver 4044 to transmit one or more portions of high-resolution sensor data 4047 to autonomous vehicle service platform 4001 for visual presentation of the physical environment on display 4009. For example, all or substantially all of the lidar data can be transmitted, although any amount less than all of the lidar data can be transmitted. The sensor-based data 4002 may be used to generate a three-dimensional view in real time (or near real time) so that the teleoperator 4008 can visually identify changes in the map data. As shown, a recently placed traffic cone 4011 is identified as being the cause of the varying data or difference between a portion of map data 4043 and local map data 4041. It should be noted that the implementations described above are only a few examples of any number of implementations of the elements shown in diagram 4000, and as such, the above description of diagram 4000 is not intended to be limiting.

[0133] FIG. 41 is a flow diagram illustrating an exemplary process of using varying amounts of locally generated map data to locate an autonomous vehicle, according to some examples. Flow 4100 begins at 4102 with locating an autonomous vehicle relative to coordinates of a global coordinate system in relation to three-dimensional map data. At 4104, variation data may be detected. That is, data representing one or more map portions of the three-dimensional map data that have varied from sensor data generated by multiple sensor types (e.g., lidar data, camera data, etc.) may be detected. In one example, blended map data may be implemented using map data from a local map and a three-dimensional map. Note that different amounts of the local map and different amounts from the three-dimensional map may be used, for example, based on the expected accuracy of the portions of the map data. In another example, at 4106, flow 4100 may implement blended map data from the local map and the three-dimensional map, and a teleoperator request may be generated at 4108. At 4110, differences between the three-dimensional map data and the sensed data (e.g., data used to generate the local map data) may be characterized, and based on the characteristics, the rate at which sensor-related data (e.g., raw sensor data, local map data, etc.) is transmitted to the autonomous vehicle service platform may be adjusted at 4112. At 4114, a three-dimensional representation of the environment from which the autonomous vehicle obtains data to depict events on a display of the teleoperator computing device is generated. Thus, the addition or absence of objects that cause differences between the map data and the locally generated map data may be visually presented to the teleoperator.

[0134] 42-43 illustrate examples of various computing platforms configured to provide various mapping-related functionality to components of an autonomous vehicle service, according to various embodiments. In some examples, computing platform 3300 may be used to implement computer programs, applications, methods, processes, algorithms, or other software to perform the techniques described above. Note that various structure and / or functionality of FIG. 33 is applicable to FIGS. 42 and 43, and thus, some elements in those figures may be discussed in the context of FIG. 33. Note further that elements illustrated in diagram 4200 of FIG. 42 and diagram 4300 of FIG. 43 may include structure and / or functionality as similarly named elements described in connection with one or more other figures, such as FIGS. 33-35, among others.

[0135] Referring to the example shown in FIG. 42, the system memory 3306 includes an autonomous vehicle service platform module 4250 and / or components thereof (e.g., mapping engine module 4252, etc.), any of which, or one or more portions thereof, may be configured to facilitate navigation for autonomous vehicle services by implementing one or more functions described herein.

[0136] 43, the system memory 3306 includes an autonomous vehicle (“AV”) module 4350 and / or components thereof (e.g., a local map generator module 4352, a blending map selection control module 4354, a communications control module 4356, etc.) and may be implemented within, for example, an autonomous vehicle 4391. In some cases, the system memory 3306 or portions thereof may be located within a mobile computing device 4390a. One or more portions of the module 4350 may be configured to facilitate navigation of an autonomous vehicle service by implementing one or more functions described herein.

[0137] Although the above examples have been described in some detail for purposes of clarity of understanding, the inventive techniques described above are not limited to the details provided. There are many alternative ways of implementing the inventive techniques described above. The disclosed examples are illustrative and not limiting.

Claims

1. receiving, via sensors associated with an autonomous vehicle traveling on a road network, sensor data related to an environment in which the autonomous vehicle is traveling, the sensor data including first sensor data generated by a first type of sensor and second sensor data generated by a second type of sensor; aligning a plurality of portions of the first sensor data representing the environment based at least in part on trajectory data associated with the autonomous vehicle to obtain aligned first sensor data; aligning the portions of the second sensor data representing the environment based at least in part on the trajectory data to obtain aligned second sensor data; refining the alignment of the aligned first sensor data and the aligned second sensor data; designating a location of the autonomous vehicle by generating position graph data and performing one or more loop closure processes to identify closed loops and detect one or more portions of the position graph data over which the autonomous vehicle has previously traveled; generating local map data based on the position graph data, the aligned first sensor data, and the aligned second sensor data; refining the local map data to reduce artifacts in the local map data to obtain refined map data; transmitting the refined map data for use by additional autonomous vehicles, the additional autonomous vehicles configured to be controlled based at least in part on the refined map data; A method comprising:

2. 2. The method of claim 1, further comprising receiving global map data; and comparing the local map data with the global map data to determine whether portions of the local map data change relative to the global map data.

3. 3. The method of claim 2, further comprising detecting changes in the local map data relative to the global map data, wherein detecting changes comprises using a statistical change detection strategy to detect portions of the local map data that have changed relative to the global map data.

4. 4. The method of claim 3, wherein using the statistical change detection strategy comprises detecting portions of the local map data that are associated with the global map data that change by identifying boundaries over one or more iterations of a deep learning computation.

5. 4. The method of claim 3, further comprising verifying the change in the local map data relative to the global map data by transmitting map tiles associated with the change to a simulator computing device and performing a simulation based at least in part on the change.

6. The method of claim 5 , further comprising, after verification, receiving, at the autonomous vehicle, updated global map data incorporating the changes.

7. receiving global map data; determining whether portions of the local map data change relative to the global map data based on comparing the local map data to the global map data; transmitting a relatively small amount of data based on determining the change to provide an alert to a teleoperator and at least monitor the autonomous vehicle traveling through an environment including minor changes between the local map data and the global map data; The method of claim 1 further comprising:

8. an autonomous vehicle; one or more sensors associated with the autonomous vehicle; one or more processors; one or more computer-readable media storing computer-executable instructions; the computer-executable instructions, when executed, cause the one or more processors to: receiving sensor data from the one or more sensors related to an environment in which the autonomous vehicle travels, the sensor data including first sensor data generated by a first type of sensor and second sensor data generated by a second type of sensor; aligning a plurality of portions of the first sensor data representing the environment based at least in part on trajectory data associated with the autonomous vehicle to obtain aligned first sensor data; aligning the portions of the second sensor data representing the environment based at least in part on the trajectory data to obtain aligned second sensor data; refining the alignment of the aligned first sensor data and the aligned second sensor data; generating position graph data and performing one or more loop closure processes to identify closed loops and designate a position of the autonomous vehicle by detecting one or more portions of the position graph data over which the autonomous vehicle has previously traveled; generating local map data based on the position graph data, the aligned first sensor data, and the aligned second sensor data; refining the local map data to reduce artifacts in the local map data to obtain refined map data; transmitting the refined map data for use by additional autonomous vehicles, the additional autonomous vehicles configured to be controlled based at least in part on the refined map data; A system for performing an operation including:

9. The operation is receiving global map data; and comparing the local map data with the global map data to determine whether portions of the local map data change relative to the global map data; transmitting a relatively small amount of data based on determining the change to provide an alert to a teleoperator and at least monitor the autonomous vehicle traveling through an environment including minor changes between the local map data and the global map data; and The system of claim 8 further comprising:

10. The operation is receiving global map data; determining whether portions of the local map data change relative to the global map data based on comparing the local map data to the global map data; verifying the change in the local map data relative to the global map data by, based on determining the change, transmitting map tiles associated with the change to a simulator computing device and performing a simulation based at least in part on the change; The system of claim 8 further comprising:

Images