Adaptive Mapping for Navigating an Autonomous Vehicle in Response to Changes in the Physical Environment

Abstract

Providing updates locally and / or remotely to maps, such as three-dimensional (3D) maps, for navigating a vehicle adapted to changes in the environment through which the vehicle traverses. The method includes accessing subsets of multiple types of sensor data, aligning the subsets of sensor data with respect to a global coordinate system based on the multiple types of sensor data, and generating datasets of three-dimensional map data. The method further includes detecting changes in data for at least two datasets of three-dimensional map data and applying the changes in data to form updated three-dimensional map data. The changes in data represent state changes in an environment in which the sensor data is sensed. The state changes in the environment may be associated with the presence or absence of an object located therein.

Description

Technical Field

[0001] Various embodiments generally relate to autonomous vehicles, and associated mechanical, electrical, and electronic hardware, computer software and systems, and wired and wireless network communications for providing a fleet of autonomous vehicles as a service. More specifically, the systems, devices, and methods are configured to provide updates locally (e.g., at the origin location of the autonomous vehicle), remotely, or both, to a map such as a three-dimensional (“3D”) map for navigating one or more of these vehicles adapted to changes in the environment through which the vehicle passes.

Background Art

[0002] Cross-Reference to Related Applications This PCT international application is a continuation of U.S. application Ser. No. 14 / 932,963, filed Nov. 4, 2015, entitled "AUTONOMOUS VEHICLE FLEET SERVICE AND SYSTEM" and U.S. patent application Ser. No. 14 / 932,959, filed Nov. 4, 2016, entitled "AUTONOMOUS VEHICLE FLEET SERVICE AND SYSTEM", and this application is related to U.S. patent application Ser. No. 14 / 932,966, filed Nov. 4, 2015, entitled "TELEOPERATION SYSTEM AND METHOD FOR TRAJECTORY MODIFICATION OF AUTONOMOUS VEHICLES", U.S. patent application Ser. No. 14 / 932,940, filed Nov. 4, 2015, entitled "AUTOMATED EXTRACTION OF SEMANTIC INFORMATION TO ENHANCE INCREMENTAL MAPPING MODIFICATIONS FOR ROBOTIC VEHICLES", U.S. patent application Ser. No. 14 / 756,995, filed Nov. 4, 2015, entitled "COORDINATION OF DISPATCHING AND MAINTAINING FLEET OF AUTONOMOUS VEHICLES", U.S. patent application Ser. No. 14 / 756,992, filed Nov. 4, 2015, entitled "ADAPTIVE AUTONOMOUS VEHICLE PLANNER LOGIC", U.S. patent application Ser. No. 14 / 756,991, filed Nov. 4, 2015, entitled "SENSOR-BASED OBJECT-DETECTION OPTIMIZATION FOR AUTONOMOUS VEHICLES", and U.S. patent application Ser. No. 14 / 756,996, filed Nov. 4, 2015, entitled "CALIBRATION FOR AUTONOMOUS VEHICLE OPERATION", all of which are hereby incorporated by reference in their entirety herein.

[0003] Various approaches for developing driverless vehicles mainly focus on automating conventional vehicles (e.g., manually operated motor vehicles) for the purpose of manufacturing driverless vehicles for consumers to purchase. For example, several automobile companies and related companies are modifying conventional automobiles and control mechanisms such as steering to provide consumers with the ability to own vehicles that can operate without a driver. In some approaches, conventional driverless vehicles implement driving functions that should prioritize safety under some conditions, but when the vehicle controller cannot solve certain problems that may endanger the safety of passengers, the driver is required to take control (e.g., steering, etc.).

[0004] Although functional, conventional driverless vehicles generally have several drawbacks. For example, many driverless vehicles under development have evolved from vehicles that require manual (i.e., human-controlled) steering and other similar automobile functions. Therefore, many driverless vehicles are designed based on the paradigm that a specific seat or location should be reserved in the vehicle for a licensed driver. As a result, driverless vehicles are sub-optimally designed and generally miss opportunities to simplify vehicle design and save resources (e.g., reduce the cost of manufacturing driverless vehicles). There are also other drawbacks in conventional driverless vehicles.

[0005] Other drawbacks are also present in conventional transportation services, which are, for example, not well-suited to managing vehicle inventory due to common approaches to effectively providing conventional transportation and carpooling services. In one conventional approach, passengers are required to access a mobile application to request a transportation service through a centralized service that assigns a human driver and a vehicle (e.g., one owned by an individual) to the passenger. By using vehicles owned by different owners, the maintenance of personal vehicles and safety systems remains uninspected. In another conventional approach, some entity enables carpooling of a group of vehicles by allowing registered drivers, as members, to access vehicles shared among the members. The driver has to board and alight from a shared vehicle at a particular location, which is rare and scarce in an urban environment and requires access to relatively expensive real estate (i.e., parking lots) to park the carpooled vehicle, so this approach is not well-suited to providing conventional transportation services. In the conventional approaches described above, the conventional vehicles used to provide transportation services are generally not fully utilized from an inventory perspective because the vehicle is rendered immobile when the driver leaves. Furthermore, carpooling approaches (and transportation services for vehicles owned by individuals) are generally not well-suited to restoring inventory balance to match the demand for transportation services in order to adapt to usage and general driving patterns. Also, some conventionally described vehicles with limited automated driving capabilities, since a human driver may generally be required, are not well-suited to restoring inventory balance. An example of a vehicle with limited automated driving capabilities is a vehicle designated as a Level 3 ("L3") vehicle by the National Highway Traffic Safety Administration ("NHTSA") of the United States Department of Transportation.

[0006] As another drawback, common approaches to autonomous vehicles are generally not well-suited to detecting and navigating a vehicle with respect to interactions (e.g., social interactions) between the vehicle in motion and other drivers or individuals of the vehicle. For example, some conventional approaches are not sufficiently capable of identifying associated interactions such as pedestrians, cyclists, etc., and eye contact, gestures, etc., for the purpose of addressing safety risks to occupants of autonomous vehicles and drivers of other vehicles, pedestrians, etc.

[0007] Accordingly, what is needed is a solution to facilitate the implementation of autonomous vehicles without the limitations of conventional techniques. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Various embodiments or examples (the "examples") of the present invention are disclosed in the following detailed description and the accompanying drawings.

[0009]

Figure 1

Figure 2

Figure 3A

Figure 3B

Figure 3C

Figure 3D

Figure 3E

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Figure 26A

Figure 26B

Figure 27

Figure 28

Figure 29

Figure 30

Figure 31

Figure 32

Figure 33

Figure 34

Figure 35

Figure 36

Figure 37

Figure 38

Figure 39

Figure 40

Figure 41

Figure 42

Figure 43

[0010] Various embodiments or examples may be implemented in several ways, including as a series of program instructions on a computer-readable medium such as a computer-readable storage medium or a computer network through which program instructions are transmitted via optical, electronic, or wireless communication links. Generally, the operations of the disclosed processes may be performed in any order, unless otherwise provided in the claims.

[0011] A detailed description of one or more examples is provided below along with the accompanying drawings. The detailed description is provided in relation to such examples but is not limited to any particular example. Its scope is limited only by the claims, as well as by some of its alternatives, modifications, and equivalents. To provide a complete understanding, some specific details are set forth in the following description. These details are provided for purposes of example and the described techniques can be practiced according to the claims without some or all of these specific details. Technical material known in the technical fields associated with the examples is not described in detail so as not to unnecessarily obscure the description.

[0012] FIG. 1 shows an implementation of a group of autonomous vehicles communicatively network-connected to an autonomous vehicle service platform according to some embodiments. FIG. 100 shows a group 109 of autonomous vehicles operating as a service (e.g., one or more of autonomous vehicles 109a - 109e), where each autonomous vehicle 109 is configured to automatically drive on a road network 110 and establish a communication link 192 with an autonomous vehicle service platform 101. In an example where the group 109 of autonomous vehicles constitutes a service, a user 102 can send a request 103 for an autonomous transportation means to the autonomous vehicle service platform 101 via one or more networks 106. In response, the autonomous vehicle service platform 101 can dispatch one of the autonomous vehicles 109 to autonomously transport the user 102 from a geographical location 119 to a geographical location 111. The autonomous vehicle service platform 101 may dispatch an autonomous vehicle from a station 190 to the geographical location 119, or may divert an already moving (e.g., without passengers) autonomous vehicle 109c to serve the transportation request of the user 102. The autonomous vehicle service platform 101 may be further configured to divert an autonomous vehicle 109c in motion with a passenger in response to a request from the user 102 (e.g., as a passenger). Additionally, the autonomous vehicle service platform 101 may be configured to secure an autonomous vehicle 109c in motion with a passenger for diversion to serve the request of the user 102 after an existing passenger has alighted. A plurality of autonomous vehicle service platforms 101 (not shown) and one or more stations 190 may be implemented to serve one or more autonomous vehicles 109 in relation to the road network 110. The one or more stations 190 may be configured to store, service, manage, and / or maintain the inventory of the autonomous vehicles 109 (e.g., the station 190 may include one or more computing devices implementing the autonomous vehicle service platform 101).

[0013] According to some examples, at least some of the autonomous vehicles 109a - 109e are configured as bidirectional autonomous vehicles ( "AVs") such as the bidirectional autonomous vehicle 130. The bidirectional autonomous vehicle 130 may be configured to travel in either direction, primarily along the longitudinal axis 131, although not limited to this. Thus, the bidirectional autonomous vehicle 130 may be configured to implement active lighting external to the vehicle to warn adjacent surrounding others (e.g., other drivers, pedestrians, cyclists, etc.) and of the direction in which the bidirectional autonomous vehicle 130 is traveling. For example, the active light source 136 may be implemented as active light 138a when traveling in the first direction, or as active light 138b when traveling in the second direction. The active light 138a may be implemented using a first subset of one or more colors, with an optional animation (e.g., a light pattern of variable intensity or a color that can change over time). Similarly, the active light 138b may be implemented using a second subset of one or more colors and a light pattern that may be different from that of the active light 138a. For example, the active light 138a may be implemented using white light as a "headlight", while the active light 138b may be implemented using red light as a "taillight". The active lights 138a and 138b, or portions thereof, may be configured to provide other light - related functions, such as providing a "turn signal indication" function (e.g., using yellow light). According to various examples, the logic within the autonomous vehicle 130 may be configured to adapt the active lights 138a and 138b to comply with various safety requirements and traffic regulations or laws regarding any number of jurisdictions.

[0014] In some embodiments, the bidirectional autonomous vehicle 130 may be configured to have similar structural and component elements within each quadrant, such as quadrants 194. The quadrants are shown, at least in this example, as portions of the bidirectional autonomous vehicle 130 defined by the intersection of planes 132 and 134 that pass through the vehicle to form two similar halves on each side of planes 132 and 134. Further, the bidirectional autonomous vehicle 130 can include an autonomous vehicle controller 147 that includes logic (e.g., hardware or software, or a combination thereof) configured to control several major vehicle functions, including driving control (e.g., propulsion, steering, etc.) and an active light source 136 among other functions. The bidirectional autonomous vehicle 130 may also include several sensors 139 disposed at various locations on the vehicle (other sensors are not shown).

[0015] The autonomous vehicle controller 147 may be further configured to determine the local posture (e.g., local position) of the autonomous vehicle 109 and to detect external objects with respect to the vehicle. For example, consider that the bidirectional autonomous vehicle 130 is traveling in direction 199 within the road network 110. A localizer (not shown) of the autonomous vehicle controller 147 may be able to determine the local position at the geographical location 111. To that end, the localizer may use the acquired sensor data, such as sensor data associated with the surfaces of the buildings 115 and 117, which may be compared against reference data, such as map data (e.g., 3D map data including reflection data), in order to determine the local position. Further, a perception engine (not shown) of the autonomous vehicle controller 147 may be configured to detect, classify, and predict the behavior of external objects, such as external object 112 (“tree”) and external object 114 (“pedestrian”). Classification of such external objects may broadly classify the objects as static objects, such as external object 112, and dynamic objects, such as external object 114. The localizer and the perception engine, as well as other components of the AV controller 147, cooperate to autonomously drive the autonomous vehicle 109.

[0016] According to some examples, the autonomous vehicle service platform 101 is configured to provide a remote operator service when an autonomous vehicle 109 requests remote operation. For example, consider that an autonomous vehicle controller 147 within an autonomous vehicle 109d detects an object 126 that is blocking the path 124 on the road 122 at point 191, as shown in the insert figure 120. If the autonomous vehicle controller 147 is unable to determine a path or trajectory along which the vehicle 109d can move safely with a relatively high degree of certainty, the autonomous vehicle controller 147 can send a request message 105 for a remote operation service. In response, the remote operator computing device 104 can receive instructions from the remote operator 108 to implement a strategy for successfully (and safely) navigating around the obstacle 126. Subsequently, response data 107 can be sent back to the autonomous vehicle 109d, for example, to safely straddle a set of double lines when moving along an alternative path 121. In some examples, the remote operator computing device 104 can generate a response that identifies a geographic area to be excluded from path planning. In particular, rather than providing a path to follow, the remote operator 108 can define an area or location that the autonomous vehicle should avoid.

[0017] In view of the above, the structure and / or function of the autonomous vehicle 130 and / or the autonomous vehicle controller 147, and their components, can perform real-time (or near real-time) trajectory calculations through autonomous-related operations such as localization and perception to enable the autonomous vehicle 109 to drive autonomously.

[0018] In some instances, the bi - directionality of the bi - directional autonomous vehicle 130 provides a vehicle having four quadrants 194 (or any other number of symmetric parts) that are similar or substantially similar to each other. Such symmetry reduces the design complexity and, relatively, the number of unique components or structures, thereby reducing the inventory and manufacturing complexity. For example, the drive system and wheel system may be disposed in any of those quadrants. Further, the autonomous vehicle controller 147 is configured to call a remote operation service to reduce the likelihood that the autonomous vehicle 109 is delayed during movement while solving events or issues that could otherwise affect the safety of the occupants. In some instances, the visible portion of the road network 110 represents a geographical fence area that can limit or otherwise control the movement of the autonomous vehicle 109 relative to the road network shown in FIG. 1. According to various examples, the autonomous vehicle 109, and its fleet, can be configured to operate as a level 4 (“fully autonomous driving” or L4) vehicle that can provide demand - responsive transportation means with the convenience and privacy of human movement between points while providing the efficiency of shared vehicles. In some examples, the autonomous vehicle 109, or any autonomous vehicle described herein, may be configured to omit a steering wheel, or any other mechanical means that provides manual (i.e., human - controlled) steering of the autonomous vehicle 109. Further, the autonomous vehicle 109, or any autonomous vehicle described herein, may be configured to omit a seat or location within the vehicle reserved for an occupant to engage a steering wheel, or any mechanical steering system.

[0019] Figure 2 is an example of a flowchart for monitoring a group of autonomous vehicles according to some embodiments. At 202, flow 200 begins when a group of autonomous vehicles is being monitored. It includes an autonomous vehicle controller configured to autonomously move at least one autonomous vehicle from a first geographic area to a second geographic area. At 204, data representing an event associated with the calculated confidence level of the vehicle is detected. An event can be a condition or situation that affects or has the potential to affect the operation of an autonomous vehicle. The event can be internal or external to the autonomous vehicle. For example, an obstacle blocking the road and a reduction or loss of communication can be considered events. The event can include traffic conditions or congestion and unexpected or abnormal numbers or types of external objects (or tracks) perceived by the perception engine. The event can include weather-related conditions (e.g., loss of friction due to ice or rain) or an angle of sunlight (e.g., at sunset) such as a low angle of sunlight that makes the eyes of a human driver of another vehicle dazzled. These and other conditions can be seen as events that trigger a call to a remote operator service or cause the vehicle to execute a safe stop trajectory.

[0020] At 206, data representing a subset of candidate trajectories can be received from the autonomous vehicle in response to the detection of an event. For example, a planner of the autonomous vehicle controller can calculate and evaluate a large number of trajectories (e.g., thousands or more) per unit time such as one second. In some embodiments, the candidate trajectories are a subset of trajectories that provide a relatively higher level of confidence that the autonomous vehicle can safely proceed considering the event (e.g., using an alternative route provided by a remote operator). Note that some candidate trajectories can be ranked higher than other candidate trajectories or can be associated with a higher confidence level than other candidate trajectories. According to some examples, the subset of candidate trajectories can be derived from any number of sources such as a planner, a remote operator computing device (e.g., the remote operator can determine and provide an approximate route), or can be combined as a subset of candidate trajectories. At 208, route guidance data can be identified in one or more processors. The route guidance data can be configured to assist the remote operator in selecting a guided trajectory from one or more of the candidate trajectories. In some cases, the route guidance data specifies a value indicating a confidence level or probability indicating the certainty that a particular candidate trajectory can reduce or ignore the possibility that an event can affect the operation of the autonomous vehicle. The guided trajectory as the selected candidate trajectory can be received at 210 in response to an input from the remote operator (e.g., the remote operator can select at least one candidate trajectory from a group of differently ranked candidate trajectories as the guided trajectory). The selection can be made via an operator interface that lists some candidate trajectories, for example, in order from the highest confidence level to the lowest confidence level. At 212, the selection of the candidate trajectory as the guided trajectory can be transmitted to the vehicle, which causes the vehicle to implement the guided trajectory for resolving the condition by performing the operation specified by the remote operator. Thus, the autonomous vehicle can transition from an abnormal operating state.

[0021] Figure 3A is a diagram showing examples of sensors and other autonomous vehicle components according to some examples. Figure 300 shows an internal view of a bidirectional autonomous vehicle 330 including a sensor, a signal router 345, a drive system 349, a removable battery 343, an audio generator 344 (e.g., a speaker or transducer), and autonomous vehicle (「AV」) control logic 347. The sensors shown in Figure 300 include, among other sensor types and modalities, an image capture sensor 340 (e.g., any type of light capture device or camera), an audio capture sensor 342 (e.g., any type of microphone), a radar (RADAR) device 348, a sonar device 341 (or other similar sensors including ultrasonic sensors or acoustic related sensors), and a lidar (LIDAR) device 346 (some of them such as an inertial measurement unit i.e. 「IMU」, a global positioning system (「GPS」) sensor, a sonar sensor, etc. are not shown). Note that the quadrants 350 represent the symmetry of each of the four 「quadrants」 of the bidirectional autonomous vehicle 330 (e.g., each quadrant 350 may include, among other things shown, wheels, a drive system 349, a similar steering mechanism, similar structural supports and members, etc.). As shown in Figure 3A, similar sensors may be placed at similar locations within each quadrant 350, but any other configuration may be implemented. Each wheel may be steerable individually and independently of each other. Also, the removable battery 343 may be configured to facilitate being swapped in and out rather than being charged in place, thereby reducing or ensuring that the downtime due to the need to charge the battery 343 is negligible. The autonomous vehicle controller 347a is shown as being used within the bidirectional autonomous vehicle 330, but the autonomous vehicle controller 347a is not so limited and may be implemented within a unidirectional autonomous vehicle or any other type of vehicle, whether on the ground, in the air, or on the sea.The positions, locations, orientations, amounts, and types of sensors shown and described in FIG. 3A are not intended to be limiting, so note that any number and type of sensors may be present, and any sensor may be disposed and oriented anywhere on the autonomous vehicle 330.

[0022] According to some embodiments, portions of the autonomous vehicle (“AV”) control logic 347 may be implemented using a cluster of graphics processing units (“GPUs”) that implement a framework and programming model suitable for programming the cluster of GPUs. For example, a programming language and application programming interface (“API”) model compliant with compute unified device architecture (“CUDA (trademark)”) may be used to program the GPUs. CUDA (trademark) is manufactured and maintained by NVIDIA of Santa Clara, California. Note that other programming languages, such as OpenCL, or any other parallel programming language may be implemented.

[0023] According to some embodiments, the autonomous vehicle control logic 347 may be implemented in hardware and / or software as an autonomous vehicle controller 347a shown to include a motion controller 362, a planner 364, a perception engine 366, and a localizer 368. As shown, the autonomous vehicle controller 347a is configured to receive camera data 340a, lidar data 346a, and radar data 348a, or any other range detection or position identification data such as sonar data 341a. The autonomous vehicle controller 347a is also configured to receive positioning data such as GPS data 352, IMU data 354, and other position detection data (e.g., wheel-related data such as steering angle, angular velocity, etc.). Further, the autonomous vehicle controller 347a may receive any other sensor data 356 and reference data 339. In some cases, the reference data 339 includes map data (e.g., including 3D map data, 2D map data, 4D map data (e.g., Epoch Determination)), and road network data such as route data (e.g., but not limited to, RNDF data (or similar data), MDF data (or similar data)).

[0024] The localizer 368 is configured to receive sensor data from one or more sources such as GPS data 352, wheel data, IMU data 354, lidar data 346a, camera data 340a, radar data 348a, and reference data 339 (e.g., 3D map data and route data). The localizer 368 integrates (e.g., fuses) and analyzes the data by comparing the sensor data with the map data in order to determine the local position (or location) of the bidirectional autonomous vehicle 330. According to some embodiments, the localizer 368 can generate or update the location or position of any autonomous vehicle in real-time or near real-time. Note that the localizer 368 and its functionality need not be limited to "bidirectional" vehicles and can be implemented in any vehicle of any type. Therefore, the localizer 368 (and other components of the AV controller 347a) may be implemented in a "unidirectional" vehicle or any non-autonomous vehicle. According to some embodiments, the data describing the local position may include one or more of an x coordinate, a y coordinate, a z coordinate (or any coordinate of any coordinate system including, for example, a polar coordinate system or a cylindrical coordinate system), a yaw value, a roll value, a pitch value (e.g., an angular value), a rate (e.g., a speed), an altitude, etc.

[0025] The perception engine 366 is configured to receive sensor data from one or more sources such as rider data 346a, camera data 340a, radar data 348a, etc., and local position data. The perception engine 366 may be configured to determine the location of external objects based on the sensor data and other data. For example, the external object may be an object that is not part of the drivable surface. For example, the perception engine 366 may be able to detect and classify external objects as pedestrians, cyclists, dogs, other vehicles, etc. (e.g., the perception engine 366 is configured to classify objects according to the type of classification that can be associated with semantic information including labels). Based on the classification of these external objects, the external objects may be labeled as dynamic objects or static objects. For example, an external object classified as a tree may be labeled as a static object, while an external object classified as a pedestrian may be labeled as a dynamic object. An external object labeled as static may or may not be described in the map data. Examples of external objects that may be labeled as static include a cement wall placed across a road, a lane closure sign, a newly placed postal pole, or a trash can adjacent to the road. Examples of external objects that may be labeled as dynamic include cyclists, pedestrians, animals, other vehicles, etc. When an external object is labeled as dynamic, additional data regarding the external object may be associated with a general level of activity and speed associated with the classification type, as well as a behavioral pattern. Additional data regarding the external object may be generated by tracking the external object. Thus, the classification type may be used to predict or otherwise determine the likelihood that an external object may interfere with, for example, an autonomous vehicle traveling along a planned route. For example, an external object classified as a pedestrian may be associated with some maximum speed and an average speed (e.g., based on tracking data).The speed of a pedestrian relative to the speed of an autonomous vehicle can be used to determine whether there is a potential for a collision. Additionally, the perception engine 364 can determine the uncertainty associated with the current and future states of an object. In some examples, the uncertainty may be expressed as an estimate (or probability).

[0026] The planner 364 is configured to receive perception data from the perception engine 366 and can also include localizer data from the localizer 368. According to some examples, the perception data can include an obstacle map that designates static and dynamic objects located in the vicinity of the autonomous vehicle, while the localizer data can include a local position or location. During operation, the planner 364 generates a number of trajectories and evaluates the trajectories based at least on the location of the autonomous vehicle relative to the relative locations of external dynamic and static objects. The planner 364 selects an optimal trajectory to guide the autonomous vehicle to provide collision-free driving based on various criteria. In some examples, the planner 364 may be configured to calculate trajectories as probabilistically determined trajectories. Additionally, the planner 364 can send steering and propulsion commands (as well as deceleration or brake commands) to the motion controller 362. The motion controller 362 can then convert any of the commands, such as a steering command, a throttle or propulsion command, and a brake command, into a control signal (e.g., for application to an actuator or other mechanical interface) for implementing a change in the steering or wheel angle and / or speed 353.

[0027] Figures 3B - 3E are diagrams showing examples of autonomous vehicle adaptation to sensor field redundancy and loss of sensor fields according to several examples. FIG. 391 in FIG. 3B shows a sensor field 301a in which sensor 310a detects an object (e.g., to determine range or distance or other information). Sensor 310a can implement any type of sensor or sensor modality, but sensor 310a, as well as sensors described similarly such as sensors 310b, 310c, and 310d, may include lidar devices. Therefore, sensor fields 301a, 301b, 301c, and 301d each include a field in which a laser extends. FIG. 392 in FIG. 3C shows four overlapping sensor fields each generated by a corresponding lidar sensor 310 (not shown). As shown, portion 301 of the sensor field does not include overlapping sensor fields (e.g., a single lidar field), portion 302 of the sensor field includes two overlapping sensor fields, and portion 303 includes three overlapping sensor fields, thereby providing multiple levels of redundancy in the event that such a sensor, a lidar sensor, malfunctions.

[0028] Figure 3D shows the loss of the sensor field due to the malfunction of the operation of the lidar 309 according to some examples. The sensor field 302 in Figure 3C has been converted into a single sensor field 305, one of the sensor fields 301 in Figure 3C is lost and has become a gap 304, and three of the sensor fields 303 in Figure 3C have been converted into a sensor field 306 (i.e., limited to two overlapping fields). When the autonomous vehicle 330c is traveling in the traveling direction 396, the sensor field in front of the moving autonomous vehicle may not be as robust as that at the rear end position. According to some examples, an autonomous vehicle controller (not shown) is configured to utilize the bi-directionality of the autonomous vehicle 330c to address the loss of the sensor field in the forward region in front of the vehicle. Figure 3E shows a bi-directional operation to recover a certain robustness of the sensor field in front of the autonomous vehicle 330d. As shown, a more robust sensor field 302 coextensive with the taillight 348 is arranged at the rear of the vehicle 330d. Conveniently, the autonomous vehicle 330d performs a bi-directional operation by approaching the unpaved road 397 and switches its bi-directionality so that the taillight 348 actively switches to the other side (e.g., the trailing edge) of the autonomous vehicle 330d. As shown, the autonomous vehicle 330d recovers a robust sensor field 302 in front of the vehicle when traveling along the traveling direction 398. Further, the bi-directional operation described above eliminates the need for a more complicated operation that requires returning to a busy lane.

[0029] Figure 4 is a functional block diagram showing a system including an autonomous vehicle service platform communicatively coupled to an autonomous vehicle controller via a communication layer according to several examples. Figure 400 shows an autonomous vehicle controller (“AV”) 447 disposed within an autonomous vehicle 430, and the autonomous vehicle 430 includes several sensors 470 coupled to the autonomous vehicle controller 447. The sensors 470 include one or more lidar devices 472, one or more cameras 474, one or more radars 476, one or more Global Positioning System (“GPS”) data receivers-sensors, one or more inertial measurement units (“IMU”) 475, one or more odometry sensors 477 (e.g., wheel encoder sensors, wheel speed sensors, etc.), and any other suitable sensors 478 such as infrared cameras or sensors, hyperspectral-capable sensors, ultrasonic sensors (or any other acoustic energy-based sensors), radio frequency-based sensors, etc. In some cases, a wheel angle sensor configured to detect the steering angle of the wheel may be included as the odometry sensor 477 or a suitable sensor 478. In a non-limiting example, the autonomous vehicle controller 447 may include four or more lidars 472, sixteen or more cameras 474, and four or more radar units 476. Further, the sensors 470 may be configured to provide sensor data to components of the autonomous vehicle controller 447 and elements of the autonomous vehicle service platform 401. As shown in Figure 400, the autonomous vehicle controller 447 includes a planner 464, a motion controller 462, a localizer 468, a perception engine 466, and a local map generator 440. Note that the elements shown in Figure 400 of Figure 4 can include the structure and / or function as similarly named elements described in connection with one or more other drawings.

[0030] The localizer 468 is configured to localize (i.e., determine the local position of) an autonomous vehicle with respect to reference data that may include map data, route data (such as road network data like RNDF data), etc. In some instances, the localizer 468 is configured to identify points in space that can represent, for example, the location of the autonomous vehicle 430 with respect to features of the environmental representation. The localizer 468 is shown to include a sensor data integrator 469 that may be configured to integrate multiple subsets of sensor data (e.g., of different sensor modalities) to reduce the uncertainty associated with each individual type of sensor. According to some examples, the sensor data integrator 469 is configured to fuse sensor data (such as lidar data, camera data, radar data, etc.) to form an integrated sensor data value for determining the local position. According to some examples, the localizer 468 retrieves reference data from a reference data repository 405 that includes a map data repository 405a for storing 2D map data, 3D map data, 4D map data, etc. The localizer 468 may be configured to identify at least a subset of features in the environment for comparison with the map data to identify or otherwise confirm the position of the autonomous vehicle 430. According to some examples, the localizer 468 may be configured to identify any amount of features in the environment such that the set of features can include one or more features or all of the features. In a particular example, any amount of lidar data (e.g., most or substantially all of the lidar data) can be compared with data representing the map for localization purposes. Generally, non-matching objects resulting from the comparison of environmental features and map data can be dynamic objects such as vehicles, cyclists, pedestrians, etc. Note that the detection of dynamic objects including obstacles can be performed with or without using the map data.In particular, dynamic objects may be detected and tracked independently of the map data (i.e., in the absence of map data). In some cases, the 2D map data and 3D map data can be regarded as "global map data" or map data authenticated at a certain point in time by the autonomous vehicle service platform 401. Since the map data in the map data repository 405a can be updated and / or authenticated periodically, there may be a deviation between the map data and the actual environment in which the autonomous vehicle is positioned. Therefore, the localizer 468 can retrieve locally derived map data generated by the local map generator 440 to enhance localization. The local map generator 440 is configured to generate local map data in real time or near real time. Optionally, the local map generator 440 may receive static and dynamic object map data to enhance the accuracy of the locally generated map, for example, by ignoring dynamic objects in localization. According to at least some embodiments, the local map generator 440 may be integrated with or formed as part of the localizer 468. In at least one case, the local map generator 440 may be configured to generate a map and / or reference data based on, for example, simultaneous localization and mapping ("SLAM"), either independently or in cooperation with the localizer 468. The localizer 468 can implement a "hybrid" approach to the use of map data, whereby the logic within the localizer 468 is configured to select various amounts of map data from either the map data from the map data repository 405a or the local map data from the local map generator 440 according to the reliability of each source of the map data. Note that the localizer 468 can still use old map data in consideration of the locally generated map data.

[0031] The perception engine 466 is configured to assist, for example, the planner 464 in planning a route and generating a trajectory by identifying objects of interest in the surrounding environment in which the autonomous vehicle 430 is moving. Further, a probability can be associated with each of the objects of interest, whereby the probability can represent the likelihood that the object of interest can pose a threat to safe driving (e.g., a fast-moving motorcycle may require enhanced tracking compared to a person sitting on a bench at a bus stop while reading a newspaper). As shown, the perception engine 466 includes an object detector 442 and an object classifier 444. The object detector 442 is configured to distinguish objects from other features in the environment, and the object classifier 444 can classify the objects as either dynamic or static objects and track the locations of the dynamic or static objects relative to the autonomous vehicle 430 for planning purposes. Further, the perception engine 466 can be configured to assign to static or dynamic objects an identifier that specifies whether the object is (or has the potential to be) an obstacle that can affect path planning in the planner 464. Although not shown in FIG. 4, it should be noted that the perception engine 466 can also perform other perception-related functions such as segmentation and tracking. Examples of these will be described later.

[0032] The planner 464 is configured to generate a number of candidate trajectories to achieve the goal of reaching a destination via some available paths or routes. A trajectory evaluator 465 is configured to evaluate the candidate trajectories and identify which subset of the candidate trajectories is associated with the highest level of confidence that provides a collision-free path to the destination. Thus, the trajectory evaluator 465 can select an optimal trajectory for the command based on the relevant criteria for generating control signals for the vehicle components 450 (e.g., actuators or other mechanisms). Note that the relevant criteria may include any number of factors that define the optimal trajectory, and the selection need not be limited to reducing collisions. For example, the selection of the trajectory may be made to optimize a collision-free trajectory that complies with user experience (e.g., user comfort), and traffic regulations and laws. User experience can be optimized by accelerating and decelerating in various linear and angular directions (e.g., to reduce driving or other uncomfortable movements such as tremors). In some cases, at least part of the relevant criteria can specify which of the other criteria should be overridden or prioritized while maintaining an optimized collision-free drive. For example, when generating a trajectory in a limited situation (e.g., when crossing a double yellow line to drive alongside a cyclist, or when driving faster than the posted speed limit to match the traffic flow), legal constraints can be temporarily waived or relaxed. Thus, the control signal is configured to cause a change in propulsion and direction in the drive train and / or wheels. In this example, the motion controller 462 is configured to convert the command into a control signal (e.g., speed, wheel angle, etc.) for controlling the movement of the autonomous vehicle 430. If the trajectory evaluator 465 has insufficient information to ensure a sufficiently high level of confidence to provide an optimized collision-free drive, the planner 464 can generate a request for the remote operator 404 to seek the support of the remote operator.

[0033] The autonomous vehicle service platform 401 includes a remote operator 404 (e.g., a remote operator computing device), a reference data repository 405, a map updater 406, a vehicle data controller 408, a calibrator 409, and an offline object classifier 410. It should be noted that each element of the autonomous vehicle service platform 401 can be independently arranged or distributed and can communicate with other elements within the autonomous vehicle service platform 401. Further, the elements of the autonomous vehicle service platform 401 can independently communicate with the autonomous vehicle 430 via the communication layer 402. The map updater 406 is configured to receive map data (e.g., from a local map generator 440, a sensor 460, or any other component of the autonomous vehicle controller 447), and is further configured to detect deviations from locally generated maps of the map data in, for example, the map data repository 405a. The vehicle data controller 408 can cause the map updater 406 to update the reference data in the repository 405 and facilitate updates to 2D, 3D, and / or 4D map data. In some cases, the vehicle data controller 408 can control the rate at which local map data is received by the autonomous vehicle service platform 408 and the frequency at which the map updater 406 performs updates to the map data.

[0034] Calibrator 409 is configured to perform calibration of various sensors of the same or different types. The calibrator 409 can be configured to determine the relative position of the sensors (e.g., in Cartesian space (x, y, z)) and the orientation of the sensors (e.g., roll, pitch, and yaw). The position and orientation of sensors such as cameras, lidar sensors, radar sensors, etc. can be calibrated relative to other sensors and globally relative to the vehicle's reference frame. Offline self-calibration can also calibrate or estimate other parameters such as the vehicle inertia tensor, wheelbase, wheel radius, or road surface friction. Calibration can also, in some examples, be performed online to detect parameter changes. Also, note that calibration by calibrator 409 can include intrinsic parameters of the sensors (e.g., optical distortion, beam angle, etc.) and extrinsic parameters. In some cases, calibrator 409 can be implemented, for example, by maximizing the correlation between depth discontinuities in 3D laser data and edges in image data. Offline object classification 410 is configured to receive data such as sensor data from sensor 470 or any other component of the autonomous vehicle controller 447. According to some embodiments, the offline classification pipeline of offline object classifier 410 can be configured to pre-group objects and annotate them (e.g., manually by a human and / or automatically using an offline labeling algorithm), and further configured to train an online classifier (e.g., object classifier 444) that can provide real-time classification of object types during online autonomous operation.

[0035] Figure 5 is an example of a flowchart for controlling an autonomous vehicle according to some embodiments. At 502, flow 500 begins when sensor data from a plurality of modality sensors in the autonomous vehicle is received, for example, by an autonomous vehicle controller. One or more subsets of the sensor data may be integrated to produce fused data, for example, to improve an estimate. In some examples, at 504, sensor streams from one or more sensors (e.g., of the same or different modalities) may be fused to form fused sensor data. In some examples, a subset of lidar sensor data and camera sensor data may be fused at 504 to facilitate localization. At 506, data representing an object may be derived in a processor based on at least two subsets of the sensor data. For example, data identifying a static or dynamic object may be derived from at least lidar and camera data (e.g., in a perception engine). At 508, detected objects that affect the planned route are determined, and at 510, a subset of trajectories is evaluated (e.g., in a planner) in response to the detected objects. At 512, a confidence level is determined that exceeds a range of acceptable confidence levels associated with the normative operation of the autonomous vehicle. Therefore, in this case, the confidence level is such that the certainty of selecting an optimized route is lower, whereby the optimized route promotes collision-free driving, complies with traffic regulations, provides a comfortable user experience (e.g., a comfortable ride), and / or may be determined according to the probability of generating candidate trajectories based on any other factors. Thus, at 514, a request for an alternative route may be sent to a remote operator computing device. The remote operator computing device may then provide the planner with an optimal trajectory for the autonomous vehicle to follow. Depending on the situation, the vehicle may also determine that a safety stop operation (e.g., safely and automatically stopping the autonomous vehicle at a location with a relatively low probability of danger) is the best course of action.The order in which the various parts of the flow diagram are shown in this flow diagram and other flow diagrams in this specification may be carried out sequentially, or in parallel with any one or more other parts of the flow diagram, or independently, or may be carried out depending on other parts of the flow diagram, and it should be noted that it is not intended to imply requirements for carrying out various functions linearly.

[0036] FIG. 6 is a diagram showing an example of the architecture of an autonomous vehicle controller according to some embodiments. FIG. 600 shows several processes including a motion controller process 662, a planner process 664, a perception process 666, a mapping process 640, and a localization process 668, some of which may generate or receive data regarding other processes. Other processes such as processes 670 and 650 can facilitate the interaction with one or more mechanical components of the autonomous vehicle. For example, the perception process 666, the mapping process 640, and the localization process 668 are configured to receive sensor data from the sensor 670, while the planner process 664 and the perception process 666 are configured to receive guidance data 606 that may include route data such as road network data. Further with respect to FIG. 600, the localization process 668 is configured to receive, among other types of map data, map data 605a (i.e., 2D map data), map data 605b (i.e., 3D map data), and local map data 642. For example, the localization process 668 may also receive other forms of map data such as 4D map data that may include, for example, epoch determination. The localization process 668 is configured to generate local position data 641 representing a local position. The local position data 641 is provided to the motion controller process 662, the planner process 664, and the perception process 666. The perception process 666 is configured to generate static and dynamic object map data 667, which may be transmitted to the planner process 664. In some examples, the static and dynamic object map data 667 may be transmitted together with other data such as semantic classification information and predicted object behavior. The planner process 664 is configured to generate trajectory data 665 that describes several trajectories generated by the planner 664.The motion controller process uses the trajectory data 665 to generate low-level commands or control signals for application to the actuator 650 to cause a change in steering angle and / or speed.

[0037] Figure 7 is a diagram showing an example of an autonomous vehicle service platform that implements redundant communication channels to maintain reliable communication with a group of autonomous vehicles according to some embodiments. Figure 700 shows an autonomous vehicle service platform 701 including a reference data generator 705, a vehicle data controller 702, an autonomous vehicle group manager 703, a remote operator manager 707, a simulator 740, and a policy manager 742. The reference data generator 705 is configured to generate and modify map data and route data (e.g., RNDF data). Further, the reference data generator 705 may be configured to access a 2D map in the 2D map data repository 720, access a 3D map in the 3D map data repository 722, and access route data in the route data repository 724. Other map representation data and repositories, such as 4D map data including epoch determination, may be implemented in some examples. The vehicle data controller 702 can be configured to perform various operations. For example, the vehicle data controller 702 may be configured to change the rate at which data is exchanged between a group of autonomous vehicles and the platform 701 based on the quality level of communication via channel 770. During periods when bandwidth is restricted, for example, data communication may be prioritized such that remote operation requests from the autonomous vehicle 730 are highly prioritized to ensure delivery. Further, depending on the bandwidth available for a particular channel, a variable level of data abstraction may be transmitted for each vehicle via channel 770. For example, in the presence of a robust network connection, all rider data (e.g., substantially all rider data, but also less may be sufficient) may be transmitted, while in the presence of a degraded or slow connection, a more simplistic or abstract depiction of the data (e.g., a bounding box with associated metadata, etc.) may be transmitted.The autonomous vehicle fleet manager 703 is configured to adjust the dispatch of the autonomous vehicles 730 to optimize a plurality of variables, including efficient use of battery power, travel time, whether the air conditioner within the autonomous vehicle 730 can be used during low battery charge states, etc., and any or all of those variables can be monitored in consideration of optimizing a cost function associated with the operation of the autonomous vehicle service. An algorithm can be implemented to analyze various variables for minimizing the cost or time of travel of the fleet of autonomous vehicles. Further, the autonomous vehicle fleet manager 703 maintains an inventory of autonomous vehicles and parts to adapt the service schedule in consideration of maximizing the uptime of the fleet.

[0038] The remote operator manager 707 is configured to manage a number of remote operator computing devices 704 that a remote operator 708 uses to provide inputs. The simulator 740 is configured to simulate the operation of one or more autonomous vehicles 730 and the interaction between the remote operator manager 707 and the autonomous vehicles 730. The simulator 740 can also simulate the operation of some sensors (including the introduction of simulated noise) disposed within the autonomous vehicles 730. Further, the simulated autonomous vehicles can be introduced into a synthetic environment, whereby an environment such as a city can be simulated so that the simulated sensors can receive simulated sensor data such as simulated laser returns. The simulator 740 can also provide other functions, including software updates and / or verification of map data. The policy manager 742 is configured to maintain a data representation policy or rule by which an autonomous vehicle should behave considering various conditions or events that the autonomous vehicle encounters while traveling on a road network. In some cases, updated policies and / or rules may be simulated in the simulator 740 to confirm the safe operation of a group of autonomous vehicles considering changes to the policy. Some of the above-described elements of the autonomous vehicle service platform 701 are further described below.

[0039] The communication channel 770 is configured to provide a network connection between the group of autonomous vehicles 730 and the autonomous vehicle service platform 701. For example, the communication channel 770 includes several different types of networks 771, 772, 773, and 774 having corresponding sub-networks (e.g., 771a~771n) to ensure a certain level of redundancy for the reliable operation of the autonomous vehicle service. For example, different types of networks within the communication channel 770 can include different cellular network providers, different types of data networks, etc., to ensure sufficient bandwidth in case of reduced or lost communication due to the malfunction of one or more of the networks 771, 772, 773, and 774.

[0040] FIG. 8 is a diagram showing an example of a messaging application configured to exchange data among various applications according to some embodiments. FIG. 800 shows a remote operator application 801 disposed within a remote operator manager and an autonomous vehicle application 830 disposed within an autonomous vehicle, whereby the remote operator application 801 and the autonomous vehicle application 830 exchange message data via a protocol that facilitates communication via various networks such as networks 871, 872, and other networks 873. According to some examples, the communication protocol is a middleware protocol implemented as a Data Distribution Service (trademark) having specifications maintained by the Object Management Group consortium. According to this communication protocol, the remote operator application 801 and the autonomous vehicle application 830 can include a message router 854 disposed within a message domain, and the message router is configured to interface with a remote operator API 852. In some examples, the message router 854 is a routing service. In some examples, a message domain 850a within the remote operator application 801 can be identified by a remote operator identifier, while the message domain 850b can be identified as a domain associated with a vehicle identifier. The remote operator API 852 within the remote operator application 801 is configured to interface with remote operator processes 803a - 803c, whereby the remote operator process 803b is associated with an autonomous vehicle identifier 804, and the remote operator process 803c is associated with an event identifier 806 (e.g., an identifier specifying an intersection that can pose a problem for planning a collision-free route). The remote operator API 852 within the autonomous vehicle application 830 is configured to interface with an autonomous vehicle operating system 840 including a sensing application 842, a perception application 844, a localization application 846, and a control application 848.In view of the above, the communication protocol described above can facilitate data exchange in order to facilitate remote operation as described herein. Further, the communication protocol described above can be adapted to provide secure data exchange between one or more autonomous vehicles and one or more autonomous vehicle service platforms. For example, the message router 854 can be configured to encrypt and decrypt messages for providing a secure interaction between, for example, the remote operator process 803 and the autonomous vehicle operating system 840.

[0041] FIG. 9 shows, by way of several examples, types of data for facilitating remote operation using the communication protocol described in FIG. 8. FIG. 900 shows a remote operator 908 interfacing with a remote operator computing device 904 coupled to a remote operator application 901 configured to exchange data via a messaging bus 972 centered on data processing implemented within one or more networks 971. The messaging bus 972 centered on data processing provides a communication link between the remote operator application 901 and the autonomous vehicle application 930. The remote operator API 962 of the remote operator application 901 is configured to receive message service configuration data 964 and route data 960 such as road network data (e.g., data such as RNDF), mission data (e.g., MDF data), and the like. Similarly, the messaging service bridge 932 is also configured to receive messaging service configuration data 934. The messaging service configuration data 934 and 964 provide configuration data for configuring the messaging service between the remote operator application 901 and the autonomous vehicle application 930. Examples of the messaging service configuration data 934 and 964 include quality of service ("QoS") configuration data implemented to configure a Data Distribution Service (trademark) application.

[0042] An example of data exchange to facilitate remote operation via a communication protocol is described as follows. Assume that obstacle data 920 is generated by the perception system of an autonomous vehicle controller. Further, planner option data 924 for notifying a subset of candidate trajectories to a remote operator is generated by a planner, and position data 926 is generated by a localizer. The obstacle data 920, the planner option data 924, and the position data 926 are sent to a messaging service bridge 932, which generates telemetry data 940 and query data 942 according to messaging service configuration data 934. Both of these are sent as telemetry data 950 and query data 952 to a remote operator application 901 via a messaging bus 972 centered on data processing. A remote operator API 962 receives the telemetry data 950 and the query data 952, and these data are processed considering route data 960 and messaging service configuration data 964. The resulting data is then presented to a remote operator 908 via a remote operator computing device 904 and / or a collaborative display (e.g., a dashboard display visible to a group of collaborating remote operators 908). The remote operator 908 considers the candidate trajectory options presented on the display of the remote operator computing device 904, selects a guided trajectory, which generates command data 982 and query response data 980. Both of these pass through the remote operator API 962 as query response data 954 and command data 956. Subsequently, the query response data 954 and the command data 956 are sent as query response data 944 and command data 946 to an autonomous vehicle application 930 via the messaging bus 972 centered on data processing. The messaging service bridge 932 receives the query response data 944 and the command data 946 and generates remote operator command data 928, which is configured to generate the trajectory selected by the remote operator for execution by the planner.Note that the messaging process described above is not intended to be limiting, and other messaging protocols may also be implemented.

[0043] FIG. 10 is a diagram showing an example of a remote operator interface that allows a remote operator to influence path planning according to some embodiments. FIG. 1000 shows an example of an autonomous vehicle 1030 communicating with an autonomous vehicle service platform 1001 that includes a remote operator manager 1007 configured to facilitate remote operation. In a first example, the remote operator manager 1007 receives data from the remote operator 1008 that requests the remote operator 1008 to preview in advance the path of the autonomous vehicle approaching a possible obstacle or an area with a low planner confidence level so that the remote operator 1008 can address the issue in advance. For illustration purposes, consider that an intersection that an autonomous vehicle is approaching can be tagged as problematic. Therefore, the user interface 1010 displays a representation 1014 of the corresponding autonomous vehicle 1030 moving along path 1012 predicted by some trajectories generated by the planner. Also displayed are other vehicles 1011 and dynamic objects 1013 such as pedestrians that can cause sufficient confusion in the planner and thereby require the support of the remote operator. The user interface 1010 also presents to the remote operator 1008 the current speed 1022, speed limit 1024, and current charge level in the battery 1026. According to some examples, the user interface 1010 may also display other data such as sensor data as obtained from the autonomous vehicle 1030. In a second example, consider that the planner 1064 has generated some trajectories that are coextensive with the path 1044 generated by the planner despite the detected unconfirmed object 1046. The planner 1064 may also generate a subset of candidate trajectories 1040, but in this example, the planner cannot proceed given the current confidence level. If the planner 1064 cannot determine an alternative path, a remote operator request may be sent. In this case, the remote operator can select one of the candidate trajectories 1040 to facilitate the autonomous vehicle 1030 to travel in accordance with the remote operator-based path 1042.

[0044] FIG. 11 is a diagram showing an example of a planner configured to call a remote operation according to several examples. FIG. 1100 shows a planner 1164 including a topography manager 1110, a route manager 1112, a path generator 1114, a trajectory evaluator 1120, and a trajectory tracker 1128. The topography manager 1110 is configured to receive map data such as 3D map data specifying topographical features or other similar map data. The topography manager 1110 is further configured to identify candidate routes based on topographical features on the route to the destination. According to various examples, the topography manager 1110 receives a 3D map generated by sensors associated with one or more autonomous vehicles in the group. The route manager 1112 is configured to receive environmental data 1103 that can include traffic-related information associated with one or more routes that can be selected as the route to the destination. The path generator 1114 receives data from the topography manager 1110 and the route manager 1112 and generates one or more paths or path segments suitable for guiding the autonomous vehicle towards the destination. Data representing the one or more paths or path segments is transmitted to the trajectory evaluator 1120.

[0045] The trajectory evaluator 1120 includes a state and event manager 1122, and the state and event manager 1122 can include a confidence level generator 1123. The trajectory evaluator 1120 further includes a guided trajectory generator 1126 and a trajectory generator 1124. Further, the planner 1164 is configured to receive policy data 1130, perception engine data 1132, and localizer data 1134.

[0046] According to some examples, the policy data 1130 can include criteria used by the planner 1164 to determine a path having a sufficient level of confidence to generate a trajectory therefrom. Examples of the policy data 1130 include a policy that specifies that trajectory generation is inhibited by a distance away from an external object (e.g., maintaining a safety buffer of 3 feet from a bicyclist, if possible), or a policy that requires that the trajectory not cross a center double yellow line, or a policy that requires that the trajectory be limited to a single lane in a four-lane road (such as concentrating generally in the lane closest to a bus stop, based on past events), and any other similar criteria specified by the policy. The perception engine data 1132 includes a map of the locations of static and dynamic objects of interest, and the localizer data 1134 includes at least a local position or location.

[0047] The state and event manager 1122 can be configured to probabilistically determine the operating state of the autonomous vehicle. For example, a first operating state (i.e., "normal operation") can describe a situation where the trajectory is collision-free, while a second operating state (i.e., "abnormal operation") can describe another situation where the confidence level associated with a possible trajectory is insufficient to guarantee collision-free driving. According to some examples, the state and event manager 1122 is configured to use the perception data 1132 to determine the state of the autonomous vehicle as being normal or abnormal. The confidence level generator 1123 can be configured to analyze the perception data 1132 to determine the state of the autonomous vehicle. For example, the confidence level generator 1123 can use semantic information associated with static and dynamic objects, and associated probability estimates, to enhance the certainty that the planner 1164 is determining a safe policy. For example, the planner 1164 can use the perception engine data 1132 that specifies the probability that an object is human or non-human to determine whether the planner 1164 is operating safely (e.g., the planner 1164 can receive a certainty having a 98% probability that the object is human and a 2% probability that the object is non-human).

[0048] Upon receiving a determination that the confidence level (e.g., based on statistics and probabilistic decision-making) falls below a threshold required for the predicted safe operation, a relatively low confidence level (e.g., a single probability score) can trigger the planner 1164 to send a request 1135 to the autonomous vehicle service platform 1101 for remote operator support. In some cases, a set of telemetry data and candidate trajectories may accompany this request. Examples of telemetry data include sensor data, location data, perception data, etc. The remote operator 1108 can send the selected trajectory 1137 to the guided trajectory generator 1126 via the remote operator computing device 1104. Thus, the selected trajectory 1137 is a trajectory formed by guidance from the remote operator. Upon receiving confirmation that there is no change in the state (e.g., the anomalous state is undecided), the guided trajectory generator 1126 passes the data to the trajectory generator 1124, and the trajectory generator 1124 causes the trajectory tracker 1128 to use the trajectory specified by the remote operator to generate a control signal 1170 (e.g., steering angle, speed, etc.) as a trajectory tracking controller. Note that the planner 1164 can trigger the transmission of the request 1135 for remote operator support before the state transitions to an anomalous state. In particular, the autonomous vehicle controller and / or its components can predict that there may be a problem with a distant obstacle and cause the planner 1164 to call for remote operation before the autonomous vehicle reaches the obstacle. Otherwise, the autonomous vehicle can cause a delay by transitioning to a safe state upon encountering an obstacle or scenario (e.g., pulling over to the shoulder and stopping). In another example, remote operation may be automatically called before the autonomous vehicle approaches a particular location where it is known to be difficult to navigate. This decision may optionally take into account other factors including time, the position of the sun, if such a situation may cause disturbances to the sensor readings and the reliability of traffic or accident data derived from various sources.

[0049] Figure 12 is an example of a flowchart configured to control an autonomous vehicle according to some embodiments. At 1202, flow 1200 begins. Data representing a subset of objects is received at a planner within the autonomous vehicle, and the subset of objects includes at least one object associated with data representing a confidence level of a classification type. For example, the perception engine data can include metadata associated with the object, whereby the metadata specifies a confidence level associated with a particular classification type. For example, a dynamic object can be classified as a "young pedestrian" with an 85% confidence level of being accurate. At 1204, localizer data can be received (e.g., at the planner). The localizer data can include map data generated locally within the autonomous vehicle. The local map data can specify the likelihood (including uncertainty) of an event occurring in a certain geographic area. An event can be a condition or situation that affects or has the potential to affect the operation of the autonomous vehicle. An event can be internal to the autonomous vehicle (e.g., a sensor that has malfunctioned or been impaired) or external (e.g., a road closure). Examples of events are described herein, such as in FIG. 2 as well as in other figures and sections. A route coextensive with the target geographic area can be determined at 1206. For example, consider an event that is the position of the sun in the sky at a time when sunlight impairs a driver's vision during rush hour traffic conditions. Therefore, traffic is expected or predicted to slow down in response to bright sunlight. Thus, the planner can pre-call for remote operation if the likelihood of an alternative route to avoid the event is low. At 1208, at the planner, a local position is determined based on local position data. At 1210, for example, based on the confidence level of the classification type and the confidence level of an event that can be based on any number of factors such as speed, position, and other state information, the operating state of the autonomous vehicle can be determined (e.g., probabilistically).For illustration purposes, consider an example where a young pedestrian is detected by an autonomous vehicle during an event where the vision of other drivers can be impaired by the sun, thereby creating a situation that is not safe for the young pedestrian. Therefore, a relatively unsafe situation can be detected as a probabilistic event that may occur (i.e., an unsafe situation where a remote operator may be called). At 1212, it is determined whether the operating state is likely to be a normative state, and based on this determination, a message is sent to the remote operator computing device requesting that it be prepared in advance for a transition to the next operating state (e.g., prepared in advance for a transition from a normative operating state to a non-normative operating state such as an unsafe operating state).

[0050] FIG. 13 is a diagram showing an example in which a planner can generate a trajectory according to some examples. FIG. 1300 includes a trajectory evaluator 1320 and a trajectory generator 1324. The trajectory evaluator 1320 includes a confidence level generator 1322 and a remote operator query manager 1329. As shown, the trajectory evaluator 1320 is coupled to a perception engine 1366 to receive static map data 1301 and data 1303 of current and predicted object states. The trajectory evaluator 1320 also receives local position data 1305 from a localizer 1368 and plan data 1307 from a global planner 1369. In one operating state (e.g., non-normative), the confidence level generator 1322 receives the static map data 1301 and the data 1303 of current and predicted object states. Based on this data, the confidence level generator 1322 can determine that the detected trajectory is associated with an unacceptable confidence level value. Therefore, the confidence level generator 1322 transmits the detected trajectory data 1309 (e.g., data including candidate trajectories) to notify the remote operator via the remote operator query manager 1329, and the remote operator query manager 1329 transmits a request 1370 for the assistance of the remote operator.

[0051] In another operating state (e.g., a normative state), static map data 1301, data 1303 of the current and predicted object states, local position data 1305, and plan data 1307 (e.g., global plan data) are received by the trajectory calculator 1325, and the trajectory calculator 1325 is configured to calculate (e.g., iteratively) a trajectory in order to determine one or more optimal routes. Next, at least one route is selected and transmitted as the data 1311 of the selected route. According to some embodiments, the trajectory calculator 1325 is configured to perform, by way of example, replanning of the trajectory. A nominal driving trajectory generator 1327 is configured to generate a trajectory in an advanced technique such as by generating a trajectory based on the receding horizon control technique. The nominal driving trajectory generator 1327 can then transmit the nominal driving trajectory path data 1372 to, for example, a trajectory tracker or a vehicle controller for performing physical changes to steering, acceleration, and other components.

[0052] FIG. 14 is a diagram showing another example of an autonomous vehicle service platform according to some embodiments. FIG. 1400 shows an autonomous vehicle service platform 1401 including a remote operator manager 1407 configured to manage the interaction and / or communication among remote operators 1408, remote operator computing devices 1404, and other components of the autonomous vehicle service platform 1401. Further with respect to FIG. 1400, the autonomous vehicle service platform 1401 includes a simulator 1440, a repository 1441, a policy manager 1442, a reference data updater 1438, a 2D map data repository 1420, a 3D map data repository 1422, and a route data repository 1424. Other map data, such as 4D map data (e.g., using epoch determination), may be implemented and stored in a repository (not shown).

[0053] The remote operator action recommendation controller 1412 includes logic configured to receive and / or control remote operation service requests via requests for remote operator assistance and autonomous vehicle (“AV”) planner data 1472 that can include telemetry data and other data. Thus, the planner data 1472 can include recommended candidate trajectories or routes that can be selected therefrom by the remote operator 1408 via the remote operator computing device 1404. According to some examples, the remote operator action recommendation controller 1412 can be configured to access other sources of recommended candidate trajectories from which an optimal trajectory should be selected. For example, candidate trajectories included within the autonomous vehicle planner data 1472 may be introduced in parallel to a simulator 1440 that is configured to simulate events or conditions experienced by the autonomous vehicle that is requesting remote operator assistance. The simulator 1440 can access map data and other data necessary to perform simulations regarding a set of candidate trajectories, whereby the simulator 1440 need not repeatedly iterate simulations to exhaustion to confirm sufficiency. Rather, the simulator 1440 can confirm the validity of candidate trajectories or, alternatively, can warn the remote operator to exercise caution in their selection.

[0054] A teleoperator interaction capture analyzer 1416 can be configured to capture a large number of teleoperator transactions or interactions for storage within a repository 1441, which can accumulate data related to some teleoperator transactions for policy analysis and generation in at least some cases. According to some embodiments, the repository 1441 can also be configured to store policy data for access by a policy manager 1442. Further, the teleoperator interaction capture analyzer 1416 can apply machine learning techniques to empirically determine how best to respond to events or conditions that trigger requests for teleoperator assistance. In some cases, the policy manager 1442 can be configured to update a particular policy or generate a new policy in response to an analysis of a large set of teleoperator interactions (e.g., after the application of machine learning techniques). The policy manager 1442 manages policies that can be viewed as rules or guidelines based on which an autonomous vehicle controller and its components operate to comply with the autonomous operation of the vehicle. In some cases, a modified or updated policy may be applied to a simulator 1440 to confirm the effectiveness of permanently releasing or implementing such a policy change.

[0055] The simulator interface controller 1414 is configured to provide an interface between the simulator 1440 and the remote operator computing device 1404. For example, sensor data from a group of autonomous vehicles is applied to the reference data update 1438 via the autonomous (「AV」) group data 1470, such that the reference data update 1438 is configured to generate updated map and route data 1439. In some embodiments, the updated map and route data 1439 may be released as an update to the data in the map data repositories 1420 and 1422, or as an update to the data in the route data repository 1424, in advance. In this case, such data may be tagged as being a 「beta version」 for which a lower threshold for requesting the remote operator service may be implemented, for example, when map tiles containing pre-updated information are used by the autonomous vehicles. Further, the updated map and route data 1439 may be introduced into the simulator 1440 to verify the updated map data. Upon receipt of the full release (e.g., at the end of beta testing), the previously lowered threshold for requesting the remote operator service is cancelled. The user interface graphics controller 1410 provides rich graphics to the remote operator 1408, such that the group of autonomous vehicles can be simulated within the simulator 1440 and the simulated group of autonomous vehicles can be accessed via the remote operator computing device 1404 as if it were real.

[0056] FIG. 15 is an example of a flowchart for controlling an autonomous vehicle according to some embodiments. At 1502, flow 1500 begins. Message data for managing a group of autonomous vehicles can be received at a remote operator computing device. The message data can indicate event attributes associated with an abnormal operating state in the context of the planned route of the autonomous vehicle. For example, an event can be characterized as a particular intersection that is problematic due to, for example, a number of pedestrians hurriedly crossing the street in violation of a traffic signal. Event attributes describe characteristics of the event such as, for example, the number of people crossing the street, the traffic delay caused by an increased number of pedestrians, and the like. At 1504, the remote operation repository can be accessed to retrieve a first subset of recommendations based on the simulated operation of the aggregated data associated with the group of autonomous vehicles. In this case, the simulator can be a source of recommendations that the remote operator can then implement. Further, the remote operation repository can also be accessed to retrieve a second subset of recommendations based on the aggregation of remote operator interactions in response to similar event attributes. In particular, the remote operator interaction capture analyzer can apply machine learning techniques to empirically determine how best to respond to events with similar attributes based on previous requests for remote operator assistance. At 1506, the first subset and the second subset of recommendations are combined to form a set of recommended policies for the autonomous vehicle. At 1508, a representation of the set of recommended policies can be visually presented on the display of the remote operator computing device. At 1510, a data signal representing a selection of the recommended policy (e.g., by the remote operator) can be detected.

[0057] FIG. 16 is a diagram of an example of an autonomous vehicle group manager that implements a group optimization manager according to several examples. FIG. 1600 shows an autonomous vehicle group manager configured to manage a group 1630 of autonomous vehicles moving within a road network 1650. The autonomous vehicle group manager 1603 is coupled to a remote operator 1608 via a remote operator computing device 1604 and is also coupled to a group management data repository 1646. The autonomous vehicle group manager 1603 is configured to receive policy data 1602, environmental data 1606, and other data. Further with respect to FIG. 1600, the group optimization manager 1620 is shown to include a movement request processor 1631, and the movement request processor 1631 includes a fleet data extractor 1632 and an autonomous vehicle dispatch optimization calculator 1634. The movement request processor 1631 is configured to process movement requests, such as those from a user 1688 requesting an autonomous vehicle service. The fleet data extractor 1632 is configured to extract data related to the autonomous vehicles within the group. The data associated with each autonomous vehicle is stored within the repository 1646. For example, the data regarding each vehicle can describe maintenance management issues, scheduled service requests, daily usage, battery charge and discharge rates, and any other data, which can be updated in real time and used for the purpose of optimizing the group of autonomous vehicles to minimize downtime. The autonomous vehicle dispatch optimization calculator 1634 is configured to analyze the extracted data and calculate an optimized usage of the group such that the next vehicle dispatched from, for example, a station 1652, guarantees to provide the minimum travel time and / or cost within the group for the autonomous vehicle service.

[0058] The swarm optimizer 1620 is shown to include a hybrid autonomous / non-autonomous vehicle processor 1640, which includes an AV / non-AV optimizer 1642 and a non-AV selector 1644. According to some examples, the hybrid autonomous / non-autonomous vehicle processor 1640 is configured to manage a hybrid swarm consisting of autonomous vehicles and vehicles driven by humans (e.g., as independent contractors). As such, the autonomous vehicle service can utilize non-autonomous vehicles to meet excess demand or in areas such as non-AV service areas 1690 that may cross geographical fences or areas of poor communication coverage. The AV / non-AV optimizer 1642 is configured to optimize the usage of the autonomous vehicle swarm and recruit non-AV drivers to the transportation service (e.g., minimizing or eliminating damage to the autonomous vehicle service). The non-AV selector 1644 includes logic for selecting the number of non-AV drivers for assistance based on calculations derived by the AV / non-AV optimizer 1642.

[0059] Figure 17 is an example of a flowchart for managing a group of autonomous vehicles according to some embodiments. At 1702, flow 1700 begins. At 1702, policy data is received. The policy data can include parameters that define how best to apply to select autonomous vehicles to serve a travel request. At 1704, group management data from a repository can be extracted. The group management data includes a subset of data regarding a pool of autonomous vehicles (e.g., the data describes whether a vehicle is ready to serve a transport request). At 1706, data representing a travel request is received. For illustrative purposes, the travel request can be for a means of transportation from a first geographic location to a second geographic location. At 1708, attributes based on the policy data are calculated to determine a subset of autonomous vehicles that are available to serve the request. For example, the attributes may include a battery charge level and the time until the next scheduled maintenance. At 1710, an autonomous vehicle is selected as a means of transportation from the first geographic location to the second geographic location, and data is generated to dispatch the autonomous vehicle to a third geographic location associated with the origin of the travel request.

[0060] FIG. 18 is a diagram showing an autonomous vehicle group manager implementing an autonomous vehicle communication link manager according to some embodiments. FIG. 1800 shows an autonomous vehicle group manager configured to manage a group 1830 of autonomous vehicles moving within a road network 1850 that coincides with a communication function stop identified as a “reduced communication area” 1880. The autonomous vehicle group manager 1803 is coupled to a remote operator 1808 via a remote operator computing device 1804. The autonomous vehicle group manager 1803 is configured to receive policy data 1802 and environmental data 1806, as well as other data. Further with respect to FIG. 1800, an autonomous vehicle communication link manager 1820 is shown to include an environment event detector 1831, a policy adaptation determinator 1832, and a movement request processor 1834. The environment event detector 1831 is configured to receive environmental data 1806 that specifies changes in the environment in which the autonomous vehicle service is being performed. For example, the environmental data 1806 can specify that the area 1880 has a degraded communication service, which may affect the autonomous vehicle service. The policy adaptation determinator 1832 can specify parameters to be applied when receiving movement requests during such events (e.g., during a communication loss). The movement request processor 1834 is configured to process movement requests taking into account the degraded communication. In this example, a user 1888 is requesting an autonomous vehicle service. Further, the movement request processor 1834 includes logic for applying an adapted policy to modify the way in which autonomous vehicles are dispatched to avoid complications arising from unsatisfactory communication.

[0061] The communication event detector 1840 includes a policy download manager 1842 and a communication-configured (``COMM-configured'') AV dispatcher 1844. The policy download manager 1842 is configured to provide an updated policy to the autonomous vehicle 1830 taking into account the reduced communication area 1880, such that the updated policy can specify a route for the autonomous vehicle to quickly exit the area 1880 if it enters the area. For example, the autonomous vehicle 1864 can receive the updated policy prior to entering the area 1880. Upon experiencing a communication loss, the autonomous vehicle 1864 implements the updated policy and selects a route 1866 to quickly exit the area 1880. The COMM-configured AV dispatcher 1844 can be configured to identify a point 1865 for parking an autonomous vehicle, which is configured as a relay device for establishing a peer-to-peer network across the area 1880. Therefore, the COMM-configured AV dispatcher 1844 is configured to dispatch an autonomous vehicle 1862 (without passengers) to park at the location 1865 for the purpose of operating as a cell tower in a peer-to-peer ad-hoc network.

[0062] FIG. 19 is an example of a flowchart for determining actions for autonomous vehicles during an event such as degraded or lost communication, according to some embodiments. At 1901, flow 1900 begins. Policy data is received, whereby the policy data defines parameters for applying to movement requests in a geographic area during an event. At 1902, one or more of the following actions may be performed. (1) Dispatch a subset of autonomous vehicles to a geographic location within a portion of that geographic location. The subset of autonomous vehicles remains at a particular geographic location and each serves as a static communication relay device or moves within the geographic area to each serve as a mobile communication relay device. (2) Perform peer-to-peer communication between a portion of a pool of autonomous vehicles associated with a portion of that geographic area. (3) Provide an event policy to the autonomous vehicles that describes a route for exiting a portion of that geographic area during the event. (4) Invoke remote operation. And (5) Recalculate the route to avoid that geographic area. After performing the actions, at 1914, the group of autonomous vehicles is monitored.

[0063] Figure 20 is a diagram showing an example of a localizer according to some embodiments. Figure 2000 includes a localizer 2068 configured to receive sensor data from a sensor 2070, such as lidar data 2072, camera data 2074, radar data 2076, and other data 2078. Further, the localizer 2068 is configured to receive reference data 2020, such as 2D map data 2022, 3D map data 2024, and 3D local map data. According to some examples, other map data, such as 4D map data 2025 and semantic map data (not shown), including corresponding data structures and repositories, may also be implemented. Further with respect to Figure 2000, the localizer 2068 includes a positioning system 2010 and a location identification system 2012, both of which are configured to receive sensor data from the sensor 2070 and reference data 2020. A location identification data integrator 2014 is configured to receive data from the positioning system 2010 and data from the location identification system 2012, whereby the location identification data integrator 2014 is configured to integrate or fuse sensor data from multiple sensors to form local location data 2052.

[0064] FIG. 21 is an example of a flowchart for generating local position data based on integrated sensor data according to some embodiments. At 2101, flow 2100 begins. At 2102, reference data is received, the reference data including three-dimensional map data. In some examples, reference data such as 3D or 4D map data may be received from one or more networks. At 2104, location data from one or more location sensors is received and placed within the location system. At 2106, positioning data from one or more positioning sensors is received and placed within the positioning system. At 2108, the location data and the positioning data are integrated. At 2110, the location data and the positioning data are integrated to form local position data that designates the geographical location of the autonomous vehicle.

[0065] Figure 22 is a diagram showing another example of a localizer according to some embodiments. Figure 2200 includes a localizer 2268, and the localizer 2268 includes a positioning system 2210 and a relative positioning system 2212 for generating data 2250 based on positioning and data 2251 based on a local location, respectively. The positioning system 2210 includes a projection processor 2254a for processing GPS data 2273, GPS datum 2211, and 3D map data 2222, among other optional data (e.g., 4D map data). The positioning system 2210 also includes an odometry processor 2254b for processing wheel data 2275 (e.g., wheel speed), vehicle model data 2213, and 3D map data 2222, among other optional data. Further, the positioning system 2210 includes an integrator processor 2254c for processing IMU data 2257, vehicle model data 2215, and 3D map data 2222, among other optional data. Similarly, the relative positioning system 2212 includes a lidar positioning processor 2254d for processing lidar data 2272, 2D tile map data 2220, 3D map data 2222, and 3D local map data 2223, among other optional data. The relative positioning system 2212 also includes a visual alignment processor 2254e for processing camera data 2274, 3D map data 2222, and 3D local map data 2223, among other optional data. Further, the relative positioning system 2212 includes a radar return processor 2254f for processing radar data 2276, 3D map data 2222, and 3D local map data 2223, among other optional data. Note that in various examples, other types of sensor data and sensors or processors, such as sonar data, may be implemented.

[0066] Furthermore, with respect to FIG. 2200, the location-based data 2250 and the relative-location-based data 2251 can be supplied to the data integrator 2266a and the location data integrator 2266, respectively. The data integrator 2266a and the location data integrator 2266 can be configured to fuse the corresponding data, whereby the location-based data 2250 can be fused in the data integrator 2266a before being fused with the relative-location-based data 2251 in the location data integrator 2266. According to some embodiments, the data integrator 2266a is formed as part of the location data integrator 2266 or is absent. Regardless, both the location-based data 2250 and the relative-location-based data 2251 can be supplied to the location data integrator 2266 for the purpose of fusing the data to generate local location data 2252. The location-based data 2250 can include unary constraint data (and uncertainties) from the projection processor 2254a, as well as binary constraint data (and uncertainties) from the odometry processor 2254b and the integrator processor 2254c. The relative-location-based data 2251 can include unary constraint data (and uncertainties) from the location processor 2254d and the visual matching processor 2254e, and optionally from the radar return processor 2254f. According to some embodiments, the location data integrator 2266 can implement a non-linear smoothing function such as a Kalman filter (e.g., a gated Kalman filter), a relative bundle adjuster, pose graph relaxation, a particle filter, a histogram filter, and the like.

[0067] FIG. 23 is a diagram showing an example of a perception engine according to some embodiments. FIG. 2300 includes a perception engine 2366, and the perception engine 2366 includes a segmentation processor 2310, an object tracker 2330, and a classifier 2360. Further, the perception engine 2366 is configured to receive, for example, local position data 2352, lidar data 2372, camera data 2374, and radar data 2376. Note that other sensor data such as sonar data may be accessed to provide the functionality of the perception engine 2366. The segmentation processor 2310 is configured to extract ground plane data and / or segment portions of an image to distinguish objects from each other and from a static image (e.g., background). In some cases, 3D blobs can be segmented to distinguish them from each other. In some examples, a blob can refer to a set of features that identify an object within a spatially reproduced environment and can be composed of elements having similar characteristics such as intensity and color (e.g., pixels of camera data, points of laser return data, etc.). In some examples, a blob can also refer to a point cloud (e.g., composed of colored laser return data) or other elements that make up an object. The object tracker 2330 is configured to perform an estimation of the movement of a blob or other segmented image portion for each frame. Further, data association is used to associate a blob at one location in a first frame at time t1 with a blob at a different location in a second frame at time t2. In some examples, the object tracker 2330 is configured to perform real-time probabilistic tracking of 3D objects such as blobs.Classifier 2360 is configured to identify an object and classify the object by classification type (such as a pedestrian, a cyclist, etc.) and by energy / activity (such as whether the object is dynamic or static), whereby the data representing the classification is described by a semantic label. According to some embodiments, a probabilistic estimation of an object category, such as classifying an object as a vehicle, a cyclist, a pedestrian, etc. with different reliabilities for each object class, can be performed. The perception engine 2366 is configured to determine perception engine data 2354, which can include a static object map and / or a dynamic object map, whereby, for example, a planner can use this information to enhance path planning. According to various examples, one or more of the segmentation processor 2310, the object tracker 2330, and the classifier 2360 can apply machine learning techniques to generate the perception engine data 2354.

[0068] FIG. 24 is an example of a flowchart for generating perception engine data according to some embodiments. Flowchart 2400 begins at 2402, where data representing the local position of the autonomous vehicle is retrieved. At 2404, position data from one or more position sensors is received, and at 2406, the features of the environment in which the autonomous vehicle is located are segmented to form segmented objects. At 2408, one or more portions of the segmented objects are spatially tracked to form at least one tracked object having motion (e.g., estimated motion). At 2410, the tracked objects are classified as being either at least static objects or dynamic objects. In some cases, static or dynamic objects can be associated with classification types. At 2412, data identifying the classified objects is generated. For example, the data identifying the classified objects can include semantic information.

[0069] Figure 25 is an example of a segmentation processor according to some embodiments. Figure 2500 shows a segmentation processor 2510 that receives lidar data from one or more lidars 2572 and camera image data from one or more cameras 2574. Local position data 2552, lidar data, and camera image data are received into a meta spin generator 2521. In some examples, the meta spin generator is configured to divide an image into distinguishable regions (e.g., clusters or groups of point clouds) based on various attributes (e.g., color, intensity, etc.), at least two or more of which can be updated simultaneously or almost simultaneously. Meta spin data 2522 is used in a segmentation processor 2523 to perform object segmentation and ground segmentation, whereby both the meta spin data 2522 and segmentation-related data from the segmentation processor 2523 are applied to a scanned differencing processor 2513. The scanned differencing processor 2513 is configured to predict the movement and / or relative speed of the segmented image portions, which can be used to identify dynamic objects at 2517. Data indicating an object having a speed detected at 2517 is optionally sent to a planner to enhance path planning decisions. Additionally, data from the scanned differencing processor 2513 can be used to approximate the location of an object to form a mapping of such an object (and optionally identify a level of movement). In some examples, an occupancy grid map 2515 can be generated. Data representing the occupancy grid map 2515 can be sent to a planner to further enhance path planning decisions (e.g., by reducing uncertainty).Regarding FIG. 2500 further, image camera data from one or more cameras 2574 is used to classify blobs in blob classifier 2520, and blob classifier 2520 also receives blob data 2524 from segmentation processor 2523. Segmentation processor 2510 can also receive unprocessed radar return data 2512 from one or more radars 2576 in order to perform segmentation in radar segmentation processor 2514 that generates radar-related blob data 2516. Regarding FIG. 25 further, segmentation processor 2510 can also receive and / or generate data 2518 of blobs being tracked that is associated with the radar data. Blob data 2516, data 2518 of blobs being tracked, data from blob classifier 2520, and blob data 2524 can be used to track an object or a portion thereof. According to some examples, one or more of the following, namely, the processor 2513 of the scanned difference, blob classification 2520, and data from radar 2576, may be optional.

[0070] Figure 26A is a diagram showing examples of an object tracker and a classifier according to various embodiments. The object tracker 2630 in FIG. 2600 is configured to receive blob data 2516, data of the blob being tracked 2518, data from the blob classifier 2520, blob data 2524, and camera image data from one or more cameras 2676. The image tracker 2633 is configured to receive camera image data from one or more cameras 2676 to generate data of the image being tracked, and the data of the image being tracked can be provided to the data association processor 2632. As shown, the data association processor 2632 is configured to receive the blob data 2516, data of the blob being tracked 2518, data from the blob classifier 2520, blob data 2524, and the tracked image data from the image tracker 2633, and is further configured to identify one or more associations between the types of data described above. The data association processor 2632 is configured, for example, to track various blob data frame by frame, in particular for estimating motion. Further, the data generated by the data association processor 2632 can be used by the track updater 2634 to update one or more tracks, or the objects being tracked. In some examples, the track updater 2634 can implement a Kalman filter, etc., to form updated data of the object being tracked that can be stored online in a track database (“DB”) 2636. Feedback data can be exchanged between the data association processor 2632 and the track database 2636 via the path 2699. In some examples, the image tracker 2633 may be optional and may be excluded. The object tracker 2630 may also use other sensor data, such as, for example, radar or sonar, and any other type of sensor data.

[0071] FIG. 26B is a diagram showing another example of an object tracker according to at least some examples. FIG. 2601 includes an object tracker 2631 that can include a structure and / or function as similarly named elements described in one or more other drawings (e.g., FIG. 26A). As shown, object tracker 2631 includes an optional alignment portion 2699 that includes a processor 2696 configured to perform object scan alignment and data fusion. Processor 2696 is further configured to store the resulting data in a 3D object database 2698.

[0072] Referring back to FIG. 26A, FIG. 2600 also includes a classifier 2660 that can include a track classification engine 2662 for generating static obstacle data 2672 and dynamic obstacle data 2674, both of which can be sent to a planner for path planning purposes. In at least one example, track classification engine 2662 is configured to determine whether an obstacle is static or dynamic and another classification type of the object (e.g., whether the object is a vehicle, pedestrian, tree, cyclist, dog, cat, paper bag, etc.). Static obstacle data 2672 can be formed as part of an obstacle map (e.g., a 2D occupancy map), and dynamic obstacle data 2674 can be formed to include a bounding box having data indicating speed and classification type. Dynamic obstacle data 2674 includes 2D dynamic obstacle map data in at least some instances.

[0073] Figure 27 is an example of a front-end processor of a perception engine according to several examples. Figure 2700 includes a grand-segmentation processor 2723a for performing grand-segmentation and an over-segmentation processor 2723b for performing "over-segmentation" according to various examples. Processors 2723a and 2723b are configured to receive optionally colored lidar data 2775. The over-segmentation processor 2723b generates data 2710 of a first blob type (e.g., relatively small blobs), and this data is provided to an aggregation classification and segmentation engine 2712 that generates data 2714 of a second blob type. The data 2714 is provided to a data association processor 2732, and the data association processor 2732 is configured to detect whether the data 2714 exists within the track database 2736. At 2740, a determination is made as to whether the data 2714 of the second blob type (e.g., a relatively large blob that may include one or more smaller blobs) is a new track. If so, a track is initialized at 2742, and if not, the tracked object data and its track stored in the track database 2736 can be extended or updated by the track update data 2742. A track classification engine 2762 is coupled to the track database 2736 to identify tracks and update / modify the tracks, for example, by adding, removing, or modifying track-related data.

[0074] FIG. 28 is a diagram showing a simulator configured to simulate an autonomous vehicle in a synthetic environment. Simulator 2840 is included in FIG. 2800 and is configured to generate a simulated environment 2803. As shown, simulator 2840 is configured to use reference data 2822 (e.g., 3D map data and / or other maps, or route data including RNDF data or similar road network data) to generate simulated geometric shapes such as simulated surfaces 2892a and 2892b within the simulated environment 2803. The simulated surfaces 2892a and 2892b can simulate the walls or sides of buildings adjacent to the road. Simulator 2840 can also generate dynamic object data either pre-generated or procedurally generated to simulate dynamic actors in the synthetic environment. An example of a dynamic actor is the simulated dynamic object 2801, which represents a simulated cyclist having a certain speed. The simulated dynamic actors can optionally respond to other static and dynamic actors within the simulated environment, including, for example, a simulated autonomous vehicle. For example, the simulated object 2801 can slow down due to other obstacles within the simulated environment 2803 rather than following a preset activation, thereby generating a more realistic simulation of the actual dynamic environment that exists in the real world.

[0075] The simulator 2840 can be configured to generate a simulated autonomous vehicle controller 2847, which includes the combined fit of a perception engine 2866, a localizer 2868, a motion controller 2862, and a planner 2864, each of which can have the functions described herein within the simulated environment 2803. The simulator 2840 can also generate a simulated interface ("I / F") 2849 for simulating data exchange with different sensor modalities and different sensor data formats. Thus, the simulated interface 2849 can simulate, for example, a software interface for packetized data from the simulated lidar sensor 2872. Additionally, the simulator 2840 can also be configured to generate a simulated autonomous vehicle 2830 that implements the simulated AV controller 2847. The simulated autonomous vehicle 2830 includes a simulated lidar sensor 2872, a simulated camera or image sensor 2874, and a simulated radar sensor 2876. In the example shown, the simulated lidar sensor 2872 can be configured to generate a simulated laser that coincides with ray tracing 2892, thereby generating a simulated sensor return 2891. Note that the simulator 2840 can simulate the addition of noise or other environmental effects on the sensor data (e.g., additional scatter or reflection that affects the simulated sensor return 2891). Furthermore, the simulator 2840 can be configured to simulate various sensor defects, including sensor malfunction, sensor miscalibration, intermittent data outages, etc.

[0076] The simulator 2840 includes a physical processor 2850 for simulating the mechanical, static, dynamic, and kinematic aspects of an autonomous vehicle for use in simulating the behavior of the simulated autonomous vehicle 2830. For example, the physical processor 2850 includes a contact mechanics module 2851 for simulating contact mechanics, a collision detection module 2852 for simulating the interaction between the simulated bodies, and a multi-body dynamics module 2854 for simulating the interaction between the simulated mechanical interactions.

[0077] Simulator 2840 also includes a simulator controller 2856 configured to control the simulation to adapt, among other things, the functionality of any synthetically generated elements of the simulated environment 2803 to determine causal relationships. Simulator 2840 includes a simulator evaluator 2858 for evaluating the performance of the synthetically generated elements of the simulated environment 2803. For example, simulator evaluator 2858 can analyze simulated vehicle commands 2880 (e.g., simulated steering angle and simulated speed) to determine whether such commands are appropriate responses to simulated activities within simulated environment 2803. Additionally, simulator evaluator 2858 can evaluate the interaction of remote operator 2808 with simulated autonomous vehicle 2830 via remote operator computing device 2804. Simulator evaluator 2858 can evaluate the effect of updated reference data 2827, including updated map tiles and route data, that can be added to guide the response of simulated autonomous vehicle 2830. Simulator evaluator 2858 can also evaluate the response of simulator AV controller 2847 when policy data 2829 is updated, deleted, or added. The above description of simulator 2840 is not intended to be limiting. Thus, simulator 2840 is configured to perform various different simulations of an autonomous vehicle with respect to a simulated environment, including both static and dynamic characteristics. For example, simulator 2840 can be used to verify changes in software versions to ensure reliability. Simulator 2840 can also be used to determine vehicle dynamics characteristics and for calibration purposes. Additionally, simulator 2840 can be used to explore the space of applicable controls and resulting trajectories to perform learning by self-simulation.

[0078] Figure 29 is an example of a flowchart for simulating various aspects of an autonomous vehicle according to some embodiments. Flowchart 2900 begins at 2902, where reference data including three-dimensional map data is received by the simulator. At 2904, dynamic object data that defines the motion patterns of classified objects can be retrieved. At 2906, a simulated environment is formed based on at least three-dimensional (“3D”) map data and dynamic object data. The simulated environment can include one or more simulated surfaces. At 2908, an autonomous vehicle including a simulated autonomous vehicle controller that forms a portion of the simulated environment is simulated. The autonomous vehicle controller can include a simulated perception engine and a simulated localizer configured to receive sensor data. At 2910, simulated sensor data is generated based on data regarding at least one simulated sensor return, and at 2912, simulated vehicle commands are generated to cause movement (e.g., vectorized propulsion) by the simulated autonomous vehicle within the synthetic environment. At 2914, the simulated vehicle commands are evaluated to determine whether the simulated autonomous vehicle behaved in accordance with the expected behavior (e.g., in accordance with a policy).

[0079] FIG. 30 is an example of a flowchart for generating map data according to some embodiments. Flowchart 3000 begins at 3002, where trajectory data is retrieved. The trajectory data can include trajectories captured over a certain time period (e.g., as logged trajectories). At 3004, at least location-specific data can be retrieved. The location-specific data can be captured over a certain time period (e.g., as logged location-specific data). At 3006, a camera or other image sensor can be implemented to generate a subset of the location-specific data. Thus, the retrieved location-specific data can include image data. At 3008, the subset of the location-specific data is aligned to identify a global location (e.g., a global position). At 3010, three-dimensional (“3D”) map data is generated based on the global position, and at 3012, the three-dimensional map data is available for implementation by, for example, a manual route data editor (including a manual road network data editor such as an RNDF editor), an automatic route data generator (including an automatic road network generator including an automatic RNDF generator), a group of autonomous vehicles, a simulator, a remote operator computing device, and any other component of an autonomous vehicle service.

[0080] Figure 31 is a diagram showing the architecture of a mapping engine according to some embodiments. Figure 3100 includes a 3D mapping engine configured to receive trajectory log data 3140, lidar log data 3172, camera log data 3174, radar log data 3176, and any other optional logged sensor data (not shown). Logic 3141 includes, among other things, a loop-closure detector 3150 configured to detect whether sensor data indicates that nearby points in space have been previously visited. Logic 3141 also includes a matching controller 3152 for aligning map data, including 3D map data in some cases, to one or more matching points. Further, logic 3141 provides data 3142 representing the state of loop closure for use by a global pose graph generator 3143 configured to generate pose graph data 3145. In some examples, the pose graph data 3145 can also be generated based on data from a matching improvement module 3146. Logic 3144 includes a 3D mapper 3154 and a lidar self-calibration unit 3156. Further, logic 3144 receives sensor data and pose graph data 3145 to generate 3D map data 3120 (or other map data such as 4D map data). In some examples, logic 3144 can implement a truncated signed distance function (“TSDF”) to fuse sensor data and / or map data to form an optimal three-dimensional map. Further, logic 3144 is configured to include texture and reflection characteristics.The 3D map data 3120 can be released for use by a manual route data editor 3160 (e.g., an editor for manipulating route data or other types of route or reference data), an automatic route data generator 3162 (e.g., logic configured to generate route data or other types of road network or reference data), a group of autonomous vehicles 3164, a simulator 3166, a remote operator computing device 3168, and any other components of the autonomous vehicle service. The mapping engine 3110 can capture semantic information from manually or automatically generated annotations and an environment equipped with sonar or devices (e.g., a smart stop lamp).

[0081] FIG. 32 is a diagram showing an autonomous vehicle application according to some embodiments. FIG. 3200 shows a mobile computing device 3203 including an autonomous service application 3240 configured to contact an autonomous vehicle service platform 3201 to configure a means of transportation for a user 3202 via an autonomous vehicle 3230. As shown, the autonomous service application 3240 can include a transport controller 3242, which may be a software application residing on a computing device (e.g., a mobile phone 3203). The transport controller 3242 is configured to receive, schedule, select, or perform operations associated with an autonomous vehicle and / or a group of autonomous vehicles that a user 3202 can use to configure a means of transportation from the user's location to a destination. For example, the user 3202 can open the application to request a vehicle 3230. The application can display a map, and the user 3202 can drop a pin, for example, to indicate the user's destination within a geographically fenced area. Alternatively, the application may display a list of nearby pre-specified pick-up locations or provide a text input field for the user to type the destination by either an address or a name.

[0082] Furthermore, for the example shown, the autonomous vehicle application 3240 can also include a user identification controller 3246 configured to detect that the user 3202 is in the geographical area near the vehicle or is nearby when the autonomous vehicle 3230 is approaching. In some situations, the user 3202 may not easily perceive or identify that the autonomous vehicle 3230 is approaching for use by the user 3203 (e.g., due to various other vehicles including trucks, cars, taxis, and other obstacles common in an urban environment). In the example, the autonomous vehicle 3230 can establish a wireless communication link 3262 (e.g., via an RF signal such as Bluetooth® including WiFi or BLE) to communicate and / or determine the spatial location of the user 3202 relative to the autonomous vehicle 3230 (e.g., using the relative direction and signal strength of a radio frequency (“RF”) signal). In some cases, the autonomous vehicle 3230 can detect the approximate geographical location of the user 3202, for example, using GPS data. A GPS receiver (not shown) of the mobile computing device 3203 can be configured to provide GPS data to the autonomous vehicle service application 3240. Thus, the user identification controller 3246 can provide the GPS data to the autonomous vehicle service platform 3201 via the link 3260, and the autonomous vehicle service platform 3201 can provide its location to the autonomous vehicle 3230 via the link 3261. Thereafter, the autonomous vehicle 3230 can determine the relative distance and / or direction of the user 3202 by comparing the user's GPS data with the location derived by the vehicle's GPS.

[0083] The autonomous vehicle 3230 can also include additional logic for identifying the presence of the user 3202, such as logic configured to generally detect the user 3202 based on the user's unique face characteristics or to implement a face detection algorithm for identifying the user 3202's identification information (e.g., name, phone number, etc.). Additionally, the autonomous vehicle 3230 can include logic for detecting a code for identifying the user 3202. Examples of such codes include specialized visual codes such as QR codes, color codes, etc., and specialized audio codes such as codes activated or recognized by voice. In some cases, the code can be an encoded security key that can be digitally transmitted to the autonomous vehicle 3230 via the link 3262 to ensure secure entry and / or exit. Further, one or more of the techniques identified above for identifying the user 3202 can be used as a secured means for granting the user 3202 the privilege of entry and exit to prevent others from entering the autonomous vehicle 3230 (e.g., to ensure that a third party does not enter an unoccupied autonomous vehicle before the user 3202 arrives). According to various examples, any other means for identifying the user 3202 and providing secured entry and exit can also be implemented in one or more of the autonomous vehicle service application 3240, the autonomous vehicle service platform 3201, and the autonomous vehicle 3230.

[0084] To assist user 3302 in reaching the requested means of transportation, autonomous vehicle 3230 can be configured to notify or otherwise alert user 3202 of its presence as autonomous vehicle 3230 approaches user 3202. For example, autonomous vehicle 3230 can activate one or more light-emitting devices 3280 (e.g., LEDs) according to a specific light pattern. In particular, a specific light pattern is generated so that user 3202 can easily perceive that autonomous vehicle 3230 has been secured to service user 3202's means of transportation request. As an example, autonomous vehicle 3230 can generate a light pattern 3290 that can be perceived by user 3202 as a "blink" or other animation of its external and internal lights in such a visual and temporal manner. The light pattern 3290 may be generated with or without an audio pattern to identify to user 3202 that this vehicle is the one reserved by the user.

[0085] According to some embodiments, the autonomous vehicle user controller 3244 can implement a software application configured to control various functions of the autonomous vehicle. Further, the application can be configured to cause the autonomous vehicle to change direction or route while moving to its initial destination. Further, the autonomous vehicle user controller 3244 can be configured to cause the built-in logic to modify the interior lighting of the autonomous vehicle 3230, for example, to perform mood lighting. The controller 3244 can also control an audio source (e.g., an external source such as Spotify, or audio stored locally on the mobile computing device 3203), select a type of ride (e.g., modify the desired acceleration and braking aggressiveness, modify the active suspension parameters to select a set of "road-handling" characteristics and implement an aggressive driving characteristic including vibration, or select a "soft-ride" where vibration is attenuated for comfort), etc. For example, the mobile computing device 3203 can be configured to also control HVAC functions such as ventilation and temperature.

[0086] Figures 33-35 illustrate examples of various computing platforms configured to provide various functions to components of an autonomous vehicle service according to various embodiments. In some examples, the computing platform 3300 can be used to implement a computer program, application, method, process, algorithm, or other software for implementing the techniques described above.

[0087] Note that the various structures and / or functions of FIG. 33 are applicable to FIGS. 34 and 35, and thus, some elements of those drawings can be discussed in the context of FIG. 33.

[0088] In some cases, computing platform 3300 may be disposed within any device, such as computing device 3390a, autonomous vehicle 3391, and / or mobile computing device 3390b, that may be disposed within one or more computing devices within the autonomous vehicle service platform.

[0089] Computing platform 3300 includes a bus 3302 or other communication mechanism for communicating information, which facilitates communication via ports on communication link 3321 with computing devices including, for example, mobile computing devices having a processor and / or communication devices, interconnecting subsystems and devices such as processor 3304, system memory 3306 (e.g., RAM, etc.), storage device 3308 (e.g., ROM, etc.), in-memory cache (which can be implemented within RAM 3306 or other parts of computing platform 3300), communication interface 3313 (e.g., Ethernet or wireless controller, Bluetooth controller, NFC logic, etc.). Processor 3304 can be implemented by one or more graphics processing units ("GPUs"), one or more central processing units ("CPUs") such as those manufactured by Intel® Corporation, or one or more virtual processors, and any combination of CPUs and virtual processors. Computing platform 3300 exchanges data representing inputs and outputs via input / output device 3301, which includes, but is not limited to, a keyboard, mouse, audio input (e.g., speech-to-text device), user interface, display, monitor, cursor, touch sensor display, LCD or LED display, and other I / O related devices.

[0090] According to some examples, the computing platform 3300 performs certain operations by a processor 3304 that executes one or more sequences of one or more instructions stored in a system memory 3306, and the computing platform 3300 can be implemented in a client-server configuration, a peer-to-peer configuration, or as any mobile computing device including, for example, a smartphone. Such instructions or data can be read from another computer-readable medium, such as a storage device 3308, into the system memory 3306. In some examples, instead of or in combination with software instructions, wired circuitry may be used for implementation. The instructions can be embodied in software or firmware. The term "computer-readable medium" refers to any tangible medium that participates in providing instructions to the processor 3304 for execution. Such a medium can take many forms, including but not limited to non-volatile media and volatile media. Non-volatile media includes, for example, optical or magnetic disks. Volatile media includes dynamic memory, such as the system memory 3306.

[0091] Common forms of computer-readable media include, for example, floppy disks, flexible disks, hard disks, magnetic tapes, or any other magnetic medium, CD-ROMs, any other optical medium, punch cards, paper tapes, any other physical medium with patterns of holes, RAM, PROM, EPROM, FLASH-EPROM, any other memory chip or cartridge, or any other medium from which a computer can read. Instructions can also be transmitted or received using a transmission medium. The term "transmission medium" can include any tangible or intangible medium capable of storing, encoding, or carrying instructions for execution by a machine, including digital or analog communication signals, or any other intangible medium facilitating the communication of such instructions. The transmission medium includes coaxial cables, copper wire, and optical fibers, including wires that include a bus 3302 for transmitting computer data signals.

[0092] In some examples, the execution of a sequence of instructions can be performed by a computing platform 3300. According to some examples, the computing platform 3300 can be coupled to any other processor by a communication link 3321 (e.g., a wired network such as a LAN, PSTN, or any wireless network including WiFi, Bluetooth®, NFC, Zig-Bee, etc. of various standards and protocols) to execute a sequence of instructions in cooperation with each other (or asynchronously with respect to each other). The computing platform 3300 can send and receive messages, data, and instructions including program code (e.g., application code) through the communication link 3321 and the communication interface 3313. The received program code may be executed by the processor 3304 when received and / or stored in the memory 3306 or other non-volatile storage device for later execution.

[0093] In the example shown, system memory 3306 can include various modules containing executable instructions for implementing the functions described herein. System memory 3306 can include an operating system (“O / S”) 3332, as well as applications 3336 and / or logical modules 3359. In the example shown in FIG. 33, system memory 3306 can include an autonomous vehicle (“AV”) controller module 3350 and / or its components (e.g., a perception engine module, a localization module, a planner module, and / or a motion controller module), any of which, or one or more portions thereof, can be configured to facilitate an autonomous vehicle service by performing one or more of the functions described herein.

[0094] Referring to the example shown in FIG. 34, system memory 3306 can include an autonomous vehicle service platform module 3450 and / or its components (e.g., a remote operator manager, a simulator, etc.), any of which, or one or more portions thereof, can be configured to facilitate the management of an autonomous vehicle service by performing one or more of the functions described herein.

[0095] Referring to the example shown in FIG. 35, system memory 3306 includes, for example, an autonomous vehicle (“AV”) module and / or its components for use in a mobile computing device. One or more portions of module 3550 can be configured to facilitate the delivery of an autonomous vehicle service by performing one or more of the functions described herein.

[0096] Referring back to FIG. 33, any of the structures and / or functions of the features described above can be implemented in software, hardware, firmware, circuitry, or combinations thereof. Note that the above structures and components and their functions may be aggregated with one or more other structures or elements. Alternatively, elements and their functions can be divided into constituent elements, if any. As software, the techniques described above may be implemented using various types of programming or formatting languages, frameworks, syntax, applications, protocols, objects, or techniques. As hardware and / or firmware, the techniques described above may be implemented using various types of programming or integrated circuit design languages, including hardware description languages such as register transfer languages ("RTL") configured to design field programmable gate arrays ("FPGA"), application specific integrated circuits ("ASIC"), or any other type of integrated circuit. According to some embodiments, the term "module" can refer to, for example, either hardware circuitry or software, or an algorithm or a portion thereof implemented in either, and / or logic. These can be modified and are not limited to the examples or descriptions provided.

[0097] In some embodiments, module 3350 of FIG. 33, module 3450 of FIG. 34, and module 3550 of FIG. 35, or one or more of their components, or any process or device described herein, can communicate with (e.g., wired or wirelessly) or be disposed within a mobile device such as a cellular phone or a mobile computing device.

[0098] In some cases, a mobile device, or any network-connected computing device (not shown) (or any process or device described herein) that communicates with one or more of the modules 3359 (module 3350 of FIG. 33, module 3450 of FIG. 34, and module 3550 of FIG. 35) or one or more of its components can provide at least some of the structure and / or function of any of the features described herein. As shown in the drawings described above, the structure and / or function of any of the features described above can be implemented in software, hardware, firmware, circuitry, or any combination thereof. It should be noted that the above structures and components and their functions may be aggregated or combined with one or more other structures or elements. Alternatively, the elements and their functions can be divided into constituent elements, if any. As software, at least some of the techniques described above may be implemented using various types of programming or format languages, frameworks, syntax, applications, protocols, objects, or techniques. For example, at least one of the elements shown in any of the drawings can represent one or more algorithms. Or, at least one of those elements can represent a logical portion that includes a portion of hardware configured to provide a configuration structure and / or function.

[0099] For example, module 3350 of FIG. 33, module 3450 of FIG. 34, and module 3550 of FIG. 35, or one or more of their components, or any process or device described herein, can be implemented in one or more computing devices (i.e., wearable devices, audio devices (such as headphones or headsets), or any mobile computing device such as a mobile phone, whether worn or carried) that include one or more processors configured to execute one or more algorithms in memory. Thus, at least some of the elements in the drawings described above can represent one or more algorithms. Or, at least one of those elements can represent a logical portion that includes a portion of hardware configured to provide a configuration structure and / or function. These can be modified and are not limited to the examples or descriptions provided.

[0100] As hardware and / or firmware, the structural techniques described above can be implemented using various types of programming languages or integrated circuit design languages, including hardware description languages such as register transfer language (「RTL」) configured to design field programmable gate arrays (「FPGA」), application specific integrated circuits (「ASIC」), multi-chip modules, or any other type of integrated circuit.

[0101] For example, module 3350 of FIG. 33, module 3450 of FIG. 34, and module 3550 of FIG. 35, or one or more of their components, or any process or device described herein, can be implemented in one or more computing devices including one or more circuits. Accordingly, at least one of the elements within the drawings described above can represent one or more components of hardware. Or, at least one of those elements can represent a logical portion including a portion of a circuit configured to provide a configuration structure and / or function.

[0102] According to some embodiments, the term "circuit" can refer to any system including some components through which current flows to perform one or more functions, and the components include individual and complex components. Examples of individual components include transistors, resistors, capacitors, inductors, diodes, etc., and examples of complex components include memories, processors, analog circuits, digital circuits, etc., including field programmable gate arrays ("FPGAs"), application specific integrated circuits ("ASICs"). Therefore, a circuit can include a system of electronic components and logical components (e.g., logic that is configured to execute instructions such as a group of executable instructions of an algorithm and is thus a component of the circuit). According to some embodiments, the term "module" can refer to, for example, either a hardware circuit or software, or an algorithm or a portion thereof implemented therein, and / or logic (i.e., a module can be implemented as a circuit). In some embodiments, an algorithm and / or the memory in which the algorithm is stored are "components" of a circuit. Accordingly, the term "circuit" can also refer to, for example, a system of components including an algorithm. These can be modified and are not limited to the examples or descriptions provided.

[0103] FIG. 36 is a diagram showing a mapping engine configured to adaptively generate mapping data for an autonomous vehicle in response to changes in a physical environment according to several examples. FIG. 3600 shows a mapping engine 3654 disposed within an autonomous vehicle service platform 3601 communicatively coupled to one or more autonomous vehicles 3630 via a communication layer (not shown). The mapping engine 3654 is configured to generate map data and adaptively modify the map data in response to changes in the physical environment through which the autonomous vehicle 3630 travels. In the example shown, the mapping engine 3654 can generate mapping data based on sensor data received from the autonomous vehicle 3630 shown as having any number of sensors or sensor devices 3604a, 3604b, and 3604c of sensor types 3602a, 3602b, and 3602c, respectively. The autonomous vehicle 3630 can include any number of other sensors or sensor devices 3604n having any other sensor type 3602n. The sensors 3604a, 3604b, 3604c, and 3604n each generate sensor data 3607a, 3607b, 3607c, and 3607n, one or more of which can be received by the mapping engine 3654 to generate map data 3659 (e.g., 2D, 3D, and / or 4D map data). The map data 3659 can be transmitted to the autonomous vehicle 3630 for storage in a map repository 3605a and for use in facilitating localization and other functionality. Specifically, the autonomous vehicle 3630 can include a localizer (not shown) that uses the map data in the map repository 3605a to determine the position and / or local position of the autonomous vehicle at any point in time, including while in transit.

[0104] In view of the foregoing, the structure and / or functionality of the mapping engine 3654, and its components, can facilitate the generation of “self-healing” maps and map data, for example, by detecting changes in a portion of the map data over time and generating an updated map (i.e., updated map data) that includes variations or changes to the physical environment through which the autonomous vehicle 3630 travels. In some implementations, the mapping engine 3654 can generate an adaptive three-dimensional model of the urban landscape adjacent to the routes and road network on which a group of autonomous vehicles travel. The 3D model of a portion of the urban landscape can identify data representing the facades or outer surfaces of objects, such as buildings (including commercial signage), trees, guardrails, barriers, streetlights, traffic signs and signals, and any other physical features that can be detected by sensors 3604a, 3604b, 3605c, and 3604n (and other surface attributes such as the shape, size, texture, color, etc. of the surface). Thus, the mapping engine 3654 can be configured to detect objects (or the absence thereof) associated with a portion of the map data, as well as changes in the objects (e.g., changes in color, size, etc.), and can be further configured to incorporate the changes to the objects into the map data to adaptively form (e.g., automatically) an updated portion of the map data. Thus, the updated portion of the map data can be stored in the map repository 3605a, among other things, to improve the accuracy of the localization function (and other autonomous vehicle controller functions, including, for example, planning) with respect to the autonomous vehicle 3630.

[0105] In some cases, the map data 3659 generated by the mapping engine 3654 can be used in combination with locally generated map data (not shown) such as that generated by a local map generator (not shown) within the autonomous vehicle 3630. For example, an autonomous vehicle controller (not shown) can detect that one or more portions of the map data in the map repository 3605a vary from one or more portions of the locally generated map data. Logic within the autonomous vehicle controller can analyze differences (e.g., variance data) in the map data to identify changes in the physical environment (e.g., addition, removal, or change in static objects). In some examples, the term "variance data" may refer to the difference between remotely generated map data and locally generated map data. Based on the changed portion of the environment, the autonomous vehicle controller can implement various proportional amounts of the map data in the map repository 3605a and the locally generated map data to optimize localization. For example, the autonomous vehicle controller can generate hybrid map data composed of both remotely generated map data and locally generated map data to optimize the determination of the position or local location of the autonomous vehicle 3630. Further, when the autonomous vehicle controller detects variance data, it can cause the transmission (at various bandwidths or data rates) of various amounts of sensor-based data or other data to the autonomous vehicle service platform 3601. For example, the autonomous vehicle service platform 3601 can receive different types of data at different data rates, e.g., based on the importance of receiving guidance from a remote operator.As another example, a subset of sensor data 3607a, 3607b, 3607c, and 3607n may be transmitted (e.g., at an appropriate data rate) to further perform one or more of the following: to modify map data to form updated map data at various levels, for example, in real time (or near real time); to evaluate and characterize differences in the map data; to propagate updated portions of the map data to other autonomous vehicles within the group; to generate a notification to a remote operator computing device in response to detecting differences in the map data; and to generate a depiction of the environment (and changed portions thereof) as sensed by various sensor devices 3604a, 3604b, 3604c, and 3604n for display at any sufficient resolution in the user interface of the remote operator computing device. Note that the examples described above are not limiting, and any other map-related functionality for managing a group of autonomous vehicles may be implemented using the mapping engine 3654 in view of changes detected in the physical environment compared to the map data.

[0106] According to some examples, sensor type 3602a, sensor type 3602b, and sensor type 3602c can each include a laser-based sensor, an image-based sensor, and a radar-based sensor. In so doing, sensors 3604a, 3604b, and 3604c can each include a lidar, a camera, and a radar device. As shown within FIG. 3600, a plurality of sensor devices (e.g., lidars) 3604a each generate data 3607a sensed based on different lasers at geographical locations. For example, each lidar 3604a can be disposed at different locations on the autonomous vehicle 3630, each of which can be differently oriented (see FIGS. 3A and 3C, both of which show different lidars with different views and sensor fields of view). Assuming the directional nature of the projected laser beam, the different laser returns of different lidars 3604a may return from a common point (or, e.g., a common set of points associated with a traffic sign) at different times. Components of the mapping engine 3654 and / or the autonomous vehicle service platform 3601 can be configured to align, map, transform, or correlate the laser returns of different lidars 3604a with respect to the common points of the laser returns from the surfaces in the environment. Components of the mapping engine 3654 and / or the autonomous vehicle service platform 3601 can also process sensor data 3607b and sensor data 3607c similarly.

[0107] In some examples, one or more sensors 3604n can include various different sensor types (“n”) 3602n to generate various different subsets of sensor data 3607n. Examples of sensors 3604n include positioning sensors such as one or more Global Positioning System (“GPS”) data receivers - sensors, one or more Inertial Measurement Units (“IMU”), one or more odometry sensors (e.g., wheel encoder sensors, wheel speed sensors, and the like), one or more wheel angle sensors, and the like that provide position and attitude data of an autonomous vehicle. Such attitude data can include one or more coordinates (e.g., x coordinate, y coordinate, and / or z coordinate), yaw value, roll value, pitch value (e.g., angular value), rate (e.g., speed), altitude, and the like.

[0108] The log data repository 3609 within the autonomous vehicle service platform 3601 is configured to receive and store subsets of sensor data 3607a, 3607b, 3607c, and 3607n, each having raw lidar data, raw camera data, raw radar data, and other raw sensor data, respectively, in at least one example. As shown in FIG. 3600, at a common point in time, or during a common time period, subsets of sensor data 3607a, 3607b, and 3607c can be stored or recorded as data set (“1”) 3610a, data set (“2”) 3610b, and data set (“n”) 3610n, or as any number of data sets. According to some examples, data sets 3610a, 3610b, and 3610n can be stored within the data structure of a log file. Further, sensor data 3607n that can be sensed simultaneously with subsets of sensor data 3607a, 3607b, and 3607c can also be stored as part of the log file for data sets 3610a, 3610b, and 3610n.

[0109] The alignment controller 3640 may be configured to receive one or more of sensor data 3607a, 3607b, 3607c, and 3607n, as well as other data 3603m. The alignment controller 3640 may also be configured to generate data representing an aligned subset of sensor data 3607a, 3607b, 3607c, and 3607n. In some cases, the sensor data 3607 can include a subset of sensor data 3607n that includes positioning data (e.g., the sensor data 3607m can include GPS, IMU, and odometry data). With respect to sensor data alignment, examples of data representing an aligned subset of sensor data include data representing at least aligned lidar data and aligned camera data. According to some examples, the alignment controller 3640 may be configured to implement an alignment algorithm to align sensor data by identifying "alignment" points that align portions or frames of lidar sensor data and portions or frames of camera data. For example, the alignment controller 3640 can map or associate laser returns from one lidar to another lidar and can map or associate pixel data from one camera to another camera. Additionally, the alignment controller 3640 can generate positioning map data, and such data can be stored in a data structure based on a pose graph model in which data specifying individual poses (e.g., local positions) can be spatially interrelated based on positioning sensor data (e.g., GPS data, IMU data, odometry data, etc.) collected from the sensors 3607n.

[0110] The mapping engine 3654 can be configured to receive the aligned sensor data (e.g., the registered sensor data) and the positioning map data (e.g., the data related to the pose graph) described above to generate a high-definition ( "HD") three-dimensional model of the urban landscape adjacent to the road network based on the integration of a subset of the sensor data 3607a, 3607b, 3607c, and 3607n. As shown in FIG. 3600, the mapping engine 3654 can include one or more of the following: an integrator 3651 for integrating sensor data, a calibrator 3652 for calibrating sensor data, a data change detector 3653 for detecting changes in portions of the map data, a tile generator 3656 for generating the formatted map data, and a data change manager 3657 for managing embodiments of the changed map data, by way of various examples.

[0111] The integrator 3651 can be configured to integrate multiple subsets of sensor data (e.g., of the same or different sensor modalities) to generate high-resolution (e.g., relatively high resolution) image data as a 3D model of the environment in which the autonomous vehicle travels, and can be further configured to reduce errors associated with the individual types of sensors. According to some examples, the integrator 3651 is configured to fuse sensor data (e.g., lidar data, camera data, radar data, etc.) to form integrated sensor data. Further, the raw sensor data sets 3610a, 3610b, and 3610n can be received from one or more autonomous vehicles 3630 to fuse a set of one or more subsets of sensor data of one or more sensor modalities from the group of autonomous vehicles 3630. By fusing the data from the raw sensor data sets 3610a, 3610b, and 3610n, the integrator 3651 can generate a 3D data set including fused sensor data such as data set ("1") 3655a and data set ("2") 3655b. The integrator 3651 can integrate or otherwise fuse at least two types of sensor data, including a subset of laser return data and a subset of image data. In some examples, the fusion of the laser and image data can include correlating the pixel data of the subset of the image data to the subset of the laser return data. Optionally, the integrator 3651 can associate the pixel data of one or more pixels to one or more laser returns, whereby the laser data can be associated with a portion of a surface within the three-dimensional tile data. The pixel data can specify one or more surface characteristics including texture, color, reflectivity, transparency, etc. According to some examples, the integrator 3651 can perform a Kalman filtering process or a variation thereof (e.g., an extended Kalman filtering process), or any other process for fusing sensor data.Integrator 3651 can also include logic for extracting or otherwise determining surface characteristics related to the surface of an object (e.g., a building, a tree, a parked automobile, etc.), or features, as well as the pose of an autonomous vehicle from which sensor data can be acquired.

[0112] Integrator 3651 can be configured to use sensor data sets 3655a and 3655b to extract data related to the surfaces of physical objects within the environment of the autonomous vehicle. Data sets 3655a and 3655b, as well as others not shown, can include fused sensor data representing three-dimensional models related to different points in time or different time intervals. Thus, data set 3655 can be used to detect whether there are changes over time in the physical environment or a portion thereof. It should be noted that in at least some implementations, Integrator 3651 can also perform a distance transform, such as a signed distance function ("SDF"), to determine one or more surfaces external to the autonomous vehicle. In one example, a truncated signed distance function ("TSDF") or the like can be implemented to identify one or more points on a surface related to a reference point (e.g., one or more distances to points on the surface of an external object related to a local position).

[0113] Integrator 3651 can be configured to generate a 3D model of an urban landscape (or features of any external object) as a probability map, whereby the map data can represent a probability distribution across one or more environmental characteristics. For example, the probability map can be formed using distances related to the pose of an autonomous vehicle or variations in laser intensity (e.g., average laser intensity or reflectivity) and infrared remittance values at points in space. A data structure for storing the map data can include several cells, for example, including intensity average values and variance values. In some examples, this data structure or any other data structure can also include several cells for storing 3D map data such as color data (e.g., RGB values or other color space values), texture data, reflectivity data, or any other surface characteristic or attribute data (e.g., specular data). Cells configured to store data related to the map can, according to some examples, be implemented as voxels or as 3D tiles.

[0114] Mapping engine 3654 and / or integrator 3651, as well as other components of mapping engine 3654, can be configured to generate a 3D map in an "offline" mode of operation. For example, mapping engine 3654 can implement an algorithm (e.g., machine learning including a deep learning algorithm) that analyzes dataset 3655 based on a dataset (e.g., static data) recorded to generate map data. However, mapping engine 3654 need not be limited to offline map generation and can also implement an "online" map generation technique where one or more portions of raw sensor data can be received in real time (or near real time) to generate map data or to identify changes to it. Mapping engine 3654 can implement logic configured to perform simultaneous localization and mapping ("SLAM") or any suitable mapping technique.

[0115] The data change detector 3653 is configured to detect changes in data sets 3655a and 3655b, which are examples of any number of data sets of 3D map data. The data change detector 3653 is also configured to identify the portions of the map data that have changed and, optionally, generate data that identifies or classifies the objects associated with the changed portions of the map data. In the example shown, some of the data sets that include data set 3655a include map data configured to generate map data conceptually shown as 3D model data 3660 (e.g., a road at time T1 that includes a portion of map data 3664). However, at time T2, the data change detector 3653 can detect that another number of data sets that include data set 3655b include data representing the presence of an external object within a portion of map data 3665 of 3D model data 3661, whereby the portion of map data 3665 does not match the portion of map data 3664 at different times. Accordingly, the data change detector 3653 can detect changes in the map data and further adaptively change the map data to include the changed map data (e.g., as updated map data).

[0116] According to some examples, the data change detector 3653 is configured to execute one or more statistical change detection algorithms to detect changes in the physical environment. Multiple time analysis techniques or other suitable algorithms may also be used. The structures of the data sets 3655a and 3655b may be implemented as cumulative data structures for indexing sensor data (e.g., its measurements) stored within the 3D map data structure. As an example, the statistical change detection algorithm may be configured to detect portions of the map data that have changed by identifying boundaries over one or more iterations of deep learning computations. Specifically, the data change detector 3653 may be configured to detect boundaries of map data portions 3664 and 3665 over time, such as over two or more data sets (e.g., over one or more passes or epochs of the application of the data sets to a statistical change detection algorithm or a deep learning algorithm). Epoch determination may also be applied, for example, to construct a 4D map and associated 4D map data. In some examples, the data change detector 3653 can classify portions of the map data as well as objects therein to identify whether the objects are static or dynamic. In some cases, dynamic objects may be excluded from map data generation.

[0117] The mapping engine 3654 is configured to provide map data 3659 to a map data repository 3605a within the reference data repository 3605. The mapping engine 3654 may be configured to apply changes in the map data to form updated three-dimensional (“3D”) map data for transmission as reference data to a reference data store (i.e., repository) within a group of autonomous vehicles. Changes in the data can represent changes in the state of the environment in which various types of sensor data are sensed. Thus, a change in the state of the environment can indicate a change in the state of an object located therein (e.g., the inclusion of data representing the presence or absence of one or more objects). In some examples, the data change manager 3657 may be configured to identify or otherwise specify (e.g., via identifier or indicator data 3658) that the map data 3658 (or a display thereof) includes a portion of the map data that has been changed. As shown, the map data 3692 stored in the map repository 3605a is associated or linked with display data (“delta data”) 3694 indicating that a relevant portion of the map data has been changed. To further add to the example shown, the display data 3694 can identify a set of traffic cones disposed within the physical environment associated with the 3D model 3661 in which the autonomous vehicle travels as the changed portion of the map data 3665.

[0118] The tile generator 3656 can be configured to generate two - dimensional or three - dimensional map tiles based on the map data from datasets 3655a and 3655b. The map tiles can be transmitted for storage within the map repository 3605a. The tile generator 3656 can generate map tiles that include indicator data for indicating that a portion of the map is an updated portion of the map data. Further, the updated map portion can be incorporated into the reference data repository 3605 within the autonomous vehicle. Thus, consider an example where the autonomous vehicle 3630 travels through a physical environment and plans to travel near a recently added object (e.g., a traffic cone) within the environment. A localizer (not shown) can access the map data associated with the changed portion of the map data (e.g., the updated portion of the map data) to locate the autonomous vehicle. Upon detecting the execution of the localization with the updated map version, the logic can perform additional processing to ensure that the use of the updated map data can be used to efficiently and safely navigate the autonomous vehicle 3630. For example, when a map tile containing the changed map data is accessed or implemented during localization, a request for monitoring or assistance to a remote operator can be generated. Note that in some examples, the changed portion of the map data may also refer to temporary map data as such data can be used in situations where, for example, there is less of it than the verified map data.

[0119] However, the changed portions of the map data can also be verified for integration into the map data, such that the state of the changed map data is noted to transition from "temporary" to "verified". To illustrate an example of verifying such data, consider that changes in the map data can be exported to a simulator computing device as updated three-dimensional map data. The simulator computing device can then simulate the performance of some of a group of autonomous vehicles in a simulated environment based on the updated three-dimensional map data. As soon as the updated three-dimensional map data is verified, the changed map portion can be incorporated to form new three-dimensional map data. The "new" three-dimensional map data can be considered reliable three-dimensional map data such that the display of the changed map data (i.e., the display of the changed map data 3694) can be removed along with the invocation of a request (e.g., an automatic request) for assistance from a remote operator.

[0120] According to some examples, the mapping engine 3654 can include and / or be implemented as a 3D mapping engine and / or mapper as shown in FIG. 31. Further, the components of the mapping engine 3654 can be combined within or without the mapping engine 3654, or otherwise be distributed. Any of the mapping engine 3654 and its components can be implemented in hardware or software, or any combination thereof. Further, the mapping engine 3654 can include any functionality and / or structure described herein, including one or more components of a perception engine for performing object detection, segmentation, and / or classification.

[0121] As a further example, consider that the alignment controller 3640 can include one or more components of the mapping engine 3110 of FIG. 31. For example, the alignment controller 3640 can include a loop-closure detector 3150, an alignment controller 3152, a global pose generator 3143, and an alignment refinement module 3146. In the example shown in FIG. 36, the autonomous vehicle service platform 3601 can implement the loop-closure detector 3150 of FIG. 31, which is configured to detect one or more portions of a position graph where the autonomous vehicle 3630 of FIG. 36 has previously traveled, as part of the alignment controller 3640 (e.g., the loop-closure detector 3150 of FIG. 31 can execute one or more loop-closure processes for identifying closed loops). The alignment controller 3152 can be configured to align or register multiple portions or frames of the same or different sensor data. For example, one or more data sets of image data can be transformed or otherwise mapped to each other, as well as to one or more data sets of laser return data and / or radar return data. The alignment controller 3152 can be configured to align a subset of laser return data, a subset of image data, and the like, based on trajectory data representing position data, to identify relative coordinates in a global coordinate system. Examples of trajectory data include GPS data, IMU data, odometry data, and the like. The global pose graph generator 3143 can be configured to generate pose graph data 3145 to specify the position of the autonomous vehicle 3630 of FIG. 36 with respect to a global coordinate system. Thus, locally detected positions in the pose graph can be referenced to the global coordinate system. For example, the global pose graph generator 3143 of FIG. 31 can be configured to form a global pose graph referenced to the global coordinate system.The global position graph can be formed based on a first type of sensor data (e.g., a subset of laser return data), a second type of sensor data (e.g., a subset of image data), and other optional sensor data (e.g., a subset of radar data). Further, the global position graph generator 3143 can also be configured to align a subset of laser return data and a subset of image data to positions related to the coordinates of a global coordinate system. The alignment refinement module 3146 is configured to refine the alignment of one or more of the captured image data, the captured laser return data, or other captured sensor data such as radar data and the like. In some examples, the alignment refinement module 3146 is configured to reduce or remove artifacts (e.g., blur artifacts or the like) of the map data, for example, following the projection of color data onto a 3D mapped surface.

[0122] FIG. 37 is a diagram showing an example of an autonomous vehicle controller that implements updated map data according to some examples. FIG. 3700 shows a mapping engine 3754 configured to generate map data 3759 that can be implemented as a three-dimensional map tile. In the example shown, the map data 3759 can also include either, or both, a portion of the changed map data (e.g., an updated portion of the map data for use with an unchanged portion of the map data), or an indication (e.g., indicator data or a pointer) that identifies an updated portion of the changed map data, i.e., changed map data 3758. Further added to FIG. 3700, the autonomous vehicle service platform 3701 can be configured to transmit map data 3786 and changed map data 3788 via network 3702. The autonomous vehicle controller 3747 uses the map data 3786 and / or the changed map data 3788 to locate the autonomous vehicle 3730. In some examples, the autonomous vehicle controller 3747 can detect that the changed map data 3788 is being accessed during the location determination. Next, the autonomous vehicle controller 3747 can generate remote operator request data 3770 to request assistance from a remote operator. The remote operator request data 3770 can also be configured such that during the location determination where an updated portion of the map data is accessed or implemented (or when the autonomous vehicle 3730 approaches or travels near a physical location associated with the updated portion of the map data), the remote operator is requested to monitor at least the performance of the autonomous vehicle 3730.

[0123] In some instances, the mapping data generated by the mapping engine 3754 can be used to generate other reference data such as route data (e.g., road network data) like that of the RNDF, mission data like that of the MDF, and other reference data that can be used to navigate a group of autonomous vehicles. As shown, a route data generator 3780 can be configured to generate route data 3782 based on the unchanged map data and / or the verified map data. Further, the route data generator 3780 can be configured to generate modified route data 3784 that can be generated using the modified and / or unverified map data. In some cases, the autonomous vehicle controller 3747 can generate remote operator request data 3770 in response to detecting the use of the modified route data 3784. Thus, the modified route data 3784 (e.g., unverified or temporary map data) can be used to navigate the autonomous vehicle with or without the assistance of the guidance data generated by the remote operator.

[0124] FIG. 38 is a flowchart illustrating an example of generating map data according to some embodiments. Flow 3800 begins at 3802. A subset of multiple types of sensor data is accessed at 3802 (e.g., within a data store or repository that can include log files). The subset of multiple types of sensor data can correspond to a group of multiple sensors or sensor devices. For example, a subset of lidar sensor data can correspond to a group of different lidar sensors where laser return data is received. At 3804, the sensor data can be aligned with respect to a global coordinate system to form aligned sensor data. For example, a positioning process or algorithm can be configured to align or position the sensor data. At 3806, a data set of three-dimensional map data can be generated based on the aligned sensor data. At 3808, changes in the map data can be detected for at least two data sets of the three-dimensional map data. The changes in the map data can be applied at 3810 to form updated three-dimensional map data. One or more updated portions of the 3D map data can be formatted as reference data for transmission to one or more vehicles within a group of autonomous vehicles. At 3812, the updated (e.g., changed) three-dimensional map data can be transmitted to at least one autonomous vehicle. Note that each portion of the flowchart can be executed independently of or dependent on other portions of the flowchart and can be executed continuously or simultaneously with any one or more other portions of the flowchart, so the order shown in this flowchart or other flowcharts in this specification is not intended to imply a need to execute various functions linearly.

[0125] FIG. 39 is a diagram showing an example of a localizer configured to implement map data and locally generated map data according to some examples. According to various examples, the localizer 3968 of the autonomous vehicle (“AV”) controller 3947 may be configured to generate local position data 3920 based on either locally generated map data 3941 or map data 3943, or a combination thereof. The local position data 3920 may include data describing the local position of the autonomous vehicle 3930, and the map data 3943 may be generated in the mapping engine 3954 of the autonomous vehicle service platform 3901. Thus, the localizer 3968 may use the map data 3943 to perform localization considering changes, deviations, or variations between the locally generated map data 3941 and the map data 3943.

[0126] FIG. 3900 shows an autonomous vehicle 3930 including an autonomous vehicle controller 3947, a local map generator 3940, and a reference data repository 3905. FIG. 3900 also shows an autonomous vehicle service platform 3901 including a mapping engine 3954 and a remote operator computing device 3904. The reference data repository 3905 may include a map store 3905a configured to store three-dimensional map data 3943, and a route data store 3905b that may be a data repository for storing route data (e.g., with or without an indication that a portion of the route data or road network data is associated with changed or updated road network data).

[0127] The local map generator 3940 can be configured to receive sensor data of multiple quantities and types, such as sensor data from sensor types 3902a, 3902b, and 3902c. According to various examples, the local map generator 3940 can be configured to locally generate map data (e.g., three-dimensional map data) in real time (or near real time) based on sensor data from sensor types 3902a, 3902b, and 3902c (e.g., from a group of lidar sensors, a group of cameras, a group of radars, etc.). The local map generator 3940 can implement logic configured to perform simultaneous localization and mapping (“SLAM”) or any suitable mapping technique. In at least some examples, the local map generator 3940 can implement an “online” map generation technique in which one or more portions of the raw sensor data from sensor types 3902a - 3902c can be received in real time (or near real time) to generate map data (or a change in identity thereto) for navigating the autonomous vehicle 3930. The local map generator 3940 can also perform a distance transform, such as a signed distance function (“SDF”), to determine the outer surface of the autonomous vehicle. In one example, a truncated signed distance function (“TSDF”) or equivalent can be performed to identify one or more points on the surface related to a reference point (e.g., one or more distances to points on the surface of an external object), whereby the TSDF function can be used to fuse sensor data and surface data to form three-dimensional local map data 3941.

[0128] The localizer 3968 can be configured to receive sensor data, as well as locally generated map data 3941 and map data 3943, in order to locate the autonomous vehicle 3930 with respect to the coordinates of a global coordinate system associated with the three-dimensional map data 3943 (or any other reference data). Also, the localizer 3968 is shown to include a variant detector 3969a and a hybrid map selection controller 3969b. The variant detector 3969a is configured to compare the locally generated map data 3941 with the map data 3943 to determine whether a portion of the map data associated with a particular surface or point in space has changed. Specifically, the variant detector 3969a can detect that data (e.g., variant data) representing two or more map portions of the local map data 3941 varies from the three-dimensional map data 3943.

[0129] When the localizer 3968 detects a varying map data portion or varying data, it may be configured to locate the autonomous vehicle 3930 using mixed map data from locally generated map data 3941 and map data 3943. In the example shown, the mixed map selection controller 3969b is configured to control whether locally generated map data 3941 or map data 3943, or a combination thereof, can be used for localization. According to some examples, for example, different amounts of locally generated map data 3941 and map data 3943 may be used based on corresponding probability distributions that can each indicate reliability or accuracy. In some examples, the mixed map selection controller 3969b may be configured to characterize the difference between one or more map portions of map data 3943 and one or more portions of local map data 3941 to form varying data. Based on the varying data, the mixed map selection controller 3969b may be configured to determine the priority of using local map data 3941 and the priority of using map data 3943, and based on the varying data, the localizer 3968 may be further configured to use a first prioritized amount of local map data 3941 and a second prioritized amount of three-dimensional map data 3943. As an example, consider an example where the variation detector 3969a detects variation data regarding some portions of the map data 3943 that vary from corresponding portions of the local map data 3941. Further consider that the local map data 3941 is determined to be more accurate for most portions of the variation data. However, at least one portion of the local map data 3941 has a relatively lower likelihood of being accurate compared to the corresponding portion of the map data 3943.In this case, the hybrid map selection controller 3969b can rely on the local map data 3941 for location identification (depending to some extent on the map data 3943), but can also rely on a specific portion of the map data 3943 for location identification (e.g., having a higher priority) rather than the corresponding portion of the local map data 3941 (e.g., having a lower priority).

[0130] Figure 40 is a diagram showing an example of a localizer configured to vary the transmission rate or amount of locally generated sensor data and / or map data according to several examples. Figure 4000 shows several autonomous vehicles including autonomous vehicles 4030a, 4030b, 4030c, and 4030n. Figure 4000 also shows an autonomous vehicle service platform 4001 including a mapping engine 4054 and remote operator logic 4004 implemented in relation to a remote operator computing device 4006 that receives data signals (e.g., user input) from a remote operator 4008. The remote operator logic 4004 can be disposed within a server computing device (not shown) or the remote operator computing device 4006. As shown, the autonomous vehicle 4030a can include an autonomous vehicle controller 4047, a reference data repository 4005 (e.g., including a map store or repository 4005a for storing map data 4046 and a route data store or repository 4005b), and a transceiver 4044 configured to exchange data between the autonomous vehicle 4030a and the autonomous vehicle service platform 4001. Further adding to Figure 4000, the autonomous vehicle controller 4070 can include a local map generator 4040 configured to generate local map data 4041 based on sensor data from different types of sensors 4002a - 4002c. The autonomous vehicle controller 4070 is also shown to include a localizer 4068 including a variance detector 4069a and a communication controller 4069b for generating local position data 4020. It should be noted that the elements shown in Figure 4000 of Figure 40 can include the structure and / or function as similarly named elements described in one or more other drawings such as Figure 39 among a number of them.

[0131] Following detection of variations between the local map data 4041 and the map data 4043 (generated by the mapping engine 4054), the communication controller 4069b may be configured to control the transceiver 4044 and the type or amount of data transmitted to the autonomous vehicle service platform 4001. Thus, according to various examples, the communication controller 4069b may be configured to provide sufficient data for the remote operation logic 4004 and / or the remote operator 4008 to select an optimal set of guidance data to resolve the detected map data variations. The communication controller 4069b may be configured to provide an optimal amount of data or data rate to conserve bandwidth. To illustrate the operation of the communication controller 4069b, consider that the variation detector 4069a detects a relatively minor or small difference between the map data 4043 and the local map data 4041. In this case, the communication controller 4069b may transmit a relatively small amount of data to provide a warning to the remote operator 4008 to prompt the remote operator to at least monitor the autonomous vehicle 4030a when traveling through an environment that includes minor changes. Further, during a degraded or low-speed data communication connection, a more simplistic or abstract data depiction may be transmitted instead of a larger amount of data (e.g., a bounding box with associated metadata).

[0132] As another example, consider that the variation detector 4069a detects a relatively moderate amount of difference between the map data 4043 and the local map data 4041. In this case, the communication controller 4069b can be configured to increase the transmission bandwidth of the transceiver 4044 to transmit one or more portions of the local map data 4041 to the autonomous vehicle service platform 4001 for evaluation by the remote operator logic 4004. In yet another example, consider that the variation detector 4069a detects a relatively large amount of difference between the map data 4043 and the local map data 4041. In this case, the communication controller 4069b can be configured to further increase the transmission bandwidth by the transceiver 4044 to transmit one or more portions of the high-resolution sensor data 4047 to the autonomous vehicle service platform 4001 for visual presentation of the physical environment on the display 4009. For example, all or substantially all of the lidar data can be transmitted, or any amount less than all of the lidar data can be transmitted. The sensor-based data 4002 can be used to generate a three-dimensional view in real time (or near real time) so that the remote operator 4008 can visually identify changes in the map data. As shown, the recently placed traffic cone 4011 is identified as the cause of the variation data or as the difference between the portion of the map data 4043 and the local map data 4041. It should be noted that the implementations described above are only a few examples of any number of implementations of the elements shown in FIG. 4000, and thus the above description of FIG. 4000 is not intended to be limiting.

[0133] FIG. 41 is a flow diagram illustrating an exemplary process of using various amounts of locally generated map data to locate an autonomous vehicle according to some examples. Flow 4100 begins at 4102, which includes locating the autonomous vehicle with respect to coordinates in a global coordinate system in relation to three-dimensional map data. At 4104, varying data may be detected. That is, data representing one or more map portions of the three-dimensional map data that have varied from sensor data (e.g., lidar data, camera data, etc.) generated by a plurality of sensor types may be detected. In one example, mixed map data may be implemented using map data from local maps and three-dimensional maps. Note that different amounts of local maps and different amounts from three-dimensional maps may be used, for example, based on the predicted accuracy of portions of the map data. In another example, at 4106, flow 4100 can implement mixed map data from local maps and three-dimensional maps, and a remote operator request may be generated at 4108. At 4110, the difference between the three-dimensional map data and the sensed data (e.g., data used to generate local map data) may be characterized, and based on the characteristics, the rate at which data related to the sensors (e.g., raw sensor data, local map data, etc.) is transmitted to the autonomous vehicle service platform may be adjusted at 4112. At 4114, a three-dimensional representation of the environment is generated in which the autonomous vehicle acquires data to draw events on the display of a remote operator computing device. Thus, the addition or absence of objects that cause a difference between the map data and the locally generated map data may be visually presented to the remote operator.

[0134] Figures 42-43 illustrate examples of various computing platforms configured to provide functionality related to various mappings to components of an autonomous vehicle service, according to various embodiments. In some examples, computing platform 3300 may be used to implement a computer program, application, method, process, algorithm, or other software for performing the techniques described above. Note that the various structures and / or functionality of FIG. 33 are applicable to FIGS. 42 and 43, and thus some elements in those figures may be discussed in the context of FIG. 33. Further note that the elements shown in FIG. 4200 of FIG. 42 and FIG. 4300 of FIG. 43 can include the structure and / or functionality as similarly named elements described in one or more other figures such as FIGS. 33-35 among others.

[0135] Referring to the example shown in FIG. 42, system memory 3306 includes autonomous vehicle service platform module 4250 and / or its components (such as mapping engine module 4252, etc.), any of which, or one or more portions thereof, can be configured to facilitate navigation for an autonomous vehicle service by implementing one or more of the functions described herein.

[0136] Referring to the example shown in FIG. 43, system memory 3306 includes autonomous vehicle ("AV") module 4350 and / or its components (such as local map generator module 4352, hybrid map selection control module 4354, communication control module 4356, etc.), and can be implemented, for example, within autonomous vehicle 4391. In some cases, system memory 3306 or a portion thereof can be disposed within mobile computing device 4390a. One or more portions of module 4350 can be configured to facilitate navigation for an autonomous vehicle service by implementing one or more of the functions described herein.

[0137] The above examples have been described in some detail for purposes of clarity of understanding, but the techniques of the present invention as described above are not limited to the details provided. There are many alternative ways of implementing the techniques of the present invention as described above. The examples disclosed are illustrative and not limiting.

Claims

1. In a computing system, receiving, from individual autonomous vehicles within a group of autonomous vehicles, multiple types of sensor data indicating driving states and objects existing in the environment around the autonomous vehicles in the group; storing the multiple types of sensor data in a data store; accessing a subset of the first type of sensor data received from a first autonomous vehicle at a first location, the subset of the first type of sensor data being derived from one or more first sensors for the first autonomous vehicle; accessing a subset of the second type of sensor data received from the first autonomous vehicle at the first location, the subset of the second type of sensor data being derived from one or more second sensors for the first autonomous vehicle; forming aligned sensor data from the subset of the first type of sensor data and the subset of the second type of sensor data; generating map data representing a probability distribution associated with one or more objects at the first location based on the aligned sensor data; in the computing system, determining whether there is a variation between the stored map data at the first location and the map data based on the probability distribution; updating the stored map data to reflect the variation; and transmitting the updated map data from the computing system to the individual autonomous vehicles within the group of autonomous vehicles, wherein at least one of the autonomous vehicles is controlled at least partially based on the updated map data. A method.

2. The step of determining whether there is a variation includes: locating the first autonomous vehicle with respect to a global coordinate system associated with the map data; detecting a difference between one or more map portions of the map data and a local map maintained by the first autonomous vehicle. The method according to claim 1.

3. characterizing the difference between the one or more map portions of the map data and the local map; further comprising updating the stored map using at least a portion of the local map and at least a portion of the one or more map portions; The method according to claim 2.

4. exporting the updated map to a simulator computing device; further comprising, at the simulator computing device, simulating the performance of a portion of the group of autonomous vehicles in the simulated environment based on the updated map; The method according to claim 1.

5. further comprising aligning the aligned sensor data with respect to a global coordinate system; The method according to claim 1.

6. An autonomous vehicle configured to autonomously drive on a road network interacting with other motor vehicles, the autonomous vehicle being a passenger vehicle and having a plurality of sensors for sensing one or more objects in the environment around the autonomous vehicle, an autonomous vehicle; A system comprising the autonomous vehicle and a computing system communicatively coupled to receive data from the autonomous vehicle and transmit commands to the autonomous vehicle, wherein the computing system, receives, from the autonomous vehicle, a plurality of types of sensor data related to the one or more objects in the environment while the autonomous vehicle is at a first location, processes the plurality of types of sensor data to determine map data representing a probability distribution associated with the one or more objects in the environment at the first location, based on the probability distribution, determines whether the sensor data indicates a change detected in the environment with respect to the one or more objects in the environment as compared to the map data of the first location, updates the map data of the first location to reflect the change detected in the environment, is programmed to transmit the updated map data to the autonomous vehicle, wherein the autonomous vehicle updates a local version of the map data using the updated map data; system.

7. The system according to claim 6, wherein the computing system is remote from the autonomous vehicle. Claim 8 The unmanned vehicle is part of the group of unmanned vehicles, and the computing system is further programmed to transmit the updated map data to a plurality of unmanned vehicles within the group. The system according to claim 6. Claim 9 The system according to claim 6, further comprising a simulator configured to simulate the performance of the group of unmanned vehicles in a simulated environment based on the updated map data. Claim 10 The system according to claim 6, further comprising a remote operator computing device configured to present the updated map data to a human remote operator. Claim 11 The computing system is further programmed to form aligned sensor data from the plurality of types of sensor data and provide the aligned sensor data related to one or more objects in the environment around the first location. The system according to claim 6.

Images