Remote real-time mapping system for autonomous vehicles

Abstract

The real-time map system can be used to propagate observations collected by autonomous vehicles operating in an environment to other autonomous vehicles to supplement the digital map used to control the autonomous vehicles. Additionally, the real-time map system can, in some cases, be used to propagate location-based teleassist triggers to autonomous vehicles operating in the environment. A location-based teleassist trigger can be generated, for example, in association with a teleassist session conducted between an autonomous vehicle proximate to a particular location and a remote teleassist system, and can be used to automatically trigger teleassist sessions for other autonomous vehicles proximate to the location and / or propagate suggested actions to the other autonomous vehicles.

Description

[Background technology]

[0001] As computing and vehicle technology continues to evolve, autonomy-related capabilities have become more powerful and widely available, enabling vehicle control in a wider variety of situations. In the case of automobiles, for example, the automotive industry has adopted the SAE International Standard J3016, which generally specifies six levels of autonomy. Vehicles without autonomy are designated Level 0, while Level 1 autonomy allows the driver to perform most vehicle functions by controlling steering or speed (but not both). Level 2 autonomy allows the vehicle to control steering, speed, and braking in limited situations (e.g., while driving on a highway), but the driver must still pay attention and be prepared to take over at any time, as well as handle any maneuvers such as lane changes and turns. From Level 3 autonomy onward, the vehicle can manage most operating variables, including monitoring the surrounding environment, but the driver must still pay attention and take over whenever the vehicle encounters a scenario it cannot handle. Level 4 autonomy provides the ability to perform driving without driver input only under specific conditions, such as certain types of roads (e.g., highways) or certain geographic areas (e.g., certain cities for which appropriate map data exists). Finally, Level 5 autonomy represents a level of autonomy that allows the vehicle to operate without driver control in any situation in which a human driver can also operate.

[0002] A fundamental challenge of any autonomy-related technology relates to gathering and interpreting information about the vehicle's surrounding environment, as well as making and implementing decisions to appropriately control the vehicle given the current environment it is operating in. Accordingly, there are ongoing efforts to improve each of these aspects, enabling autonomous vehicles to reliably handle an increasingly diverse range of situations and accommodate both expected and unforeseen conditions within an environment.

[0003] For example, special challenges are introduced in the inherently dynamic environments in which autonomous vehicles are expected to operate. For example, many autonomous vehicles rely on high-resolution digital maps that represent various static objects in the environment, including logical elements such as lanes and boundaries, as well as real-world objects or elements such as roads, curbs, buildings, trees, and signs. While attempts are typically made to maintain and update high-quality digital maps to accommodate changes occurring in the environment, the overhead associated with validating and distributing map data to fleets of autonomous vehicles can be significant. Additionally, despite rapid updates, changes to the environment may occur suddenly and not be reflected in the map data used by autonomous vehicles operating in that environment.

[0004] Additionally, while in some cases autonomous vehicle perception systems can be used to detect changed conditions in the environment (e.g., the presence of new construction elements such as cones and / or barrels), the detection range of such systems is generally limited and may be obstructed by the presence of other vehicles nearby, which may result in an undesirably short amount of time for the autonomous vehicle to react to some changed conditions. Thus, there is a continuing need in the industry for ways to improve autonomous vehicle perception of relevant objects and elements in the environment. Summary of the Invention

[0005] This disclosure relates, in part, to the use of a real-time map system to supplement a digital map used to control an autonomous vehicle by propagating observations collected by the autonomous vehicle operating in an environment to other autonomous vehicles. The real-time map system can also be used, in some cases, to propagate Location-Based Teleassist Triggers to autonomous vehicles operating in the environment. A location-based teleassist trigger may be generated, for example, in association with a teleassist session between an autonomous vehicle proximate to a particular location and a remote teleassist system, and can be used to automatically trigger teleassist sessions with other autonomous vehicles proximate to the location and / or propagate suggested actions to the other autonomous vehicles.

[0006] Thus, according to one aspect of the present invention, an autonomous vehicle control system includes one or more processors and a memory storing instructions that, when executed by the one or more processors, cause the autonomous vehicle control system to receive a digital map of a portion of an environment in which an autonomous vehicle is operating, the digital map defining a plurality of static elements within the portion of the environment, receive observation data relating to one or more observations collected from the environment from a remote real-time map system, augment the digital map with the observation data to generate an augmented digital map, and control the autonomous vehicle using the augmented digital map.

[0007] In some embodiments, the one or more processors are configured to control the autonomous vehicle using the augmented digital map by determining a trajectory of the autonomous vehicle using the augmented digital map and controlling the autonomous vehicle along the trajectory.

[0008] In some embodiments, the observational data is first observational data, and the one or more processors are further configured to receive second observational data collected using one or more sensors of the autonomous vehicle, augment the digital map with the second observational data, and the augmented digital map is augmented using the first observational data and the second observational data. In some embodiments, the one or more processors are further configured to forward the second observational data to the remote real-time mapping system for use in controlling one or more other autonomous vehicles operating in the environment.

[0009] In some embodiments, the one or more observations collected from the environment are collected by one or more other autonomous vehicles operating within the environment and forwarded to the remote real-time map system. Also, in some embodiments, the observation data defines construction elements detected in the environment. In some embodiments, the observation data defines occluded areas within the environment that restrict vehicle movement within the environment. Also, in some embodiments, the observation data defines one or more vehicle paths, each vehicle path related to detected paths of other vehicles while traveling through the environment, and the one or more processors are configured to control the autonomous vehicle using the augmented digital map by determining a movement path for the autonomous vehicle using the one or more vehicle paths.

[0010] In some embodiments, the observational data defines a suboptimal action to be taken by another autonomous vehicle while navigating the environment, and the one or more processors are configured to control the autonomous vehicle using the augmented digital map by determining a movement path for the autonomous vehicle using the suboptimal action. In some embodiments, the one or more processors are further configured to receive a position-based teleassist trigger from the remote real-time map system and determine activation of the position-based teleassist trigger based at least in part on position-based criteria associated with the position-based teleassist trigger.

[0011] Also, in some embodiments, the one or more processors are further configured to automatically initiate a teleassist session with a remote teleassist system in response to determining activation of the position-based teleassist trigger. In some embodiments, the one or more processors are further configured to control the autonomous vehicle based at least in part on a proposed action associated with the position-based teleassist trigger in response to determining activation of the position-based teleassist trigger. Also, in some embodiments, the one or more processors are configured to control the autonomous vehicle based at least in part on the proposed action associated with the position-based teleassist trigger without initiating a teleassist session with a remote teleassist system. In some embodiments, the proposed action is received from the remote real-time map system and generated from another teleassist session with another autonomous vehicle.

[0012] According to another aspect of the invention, a method of operating an autonomous vehicle using an autonomous vehicle control system may include receiving a digital map of a portion of an environment in which the autonomous vehicle is to operate, the digital map defining a plurality of static elements within the portion of the environment; receiving observation data associated with one or more observations collected from the environment from a remote real-time map system; augmenting the digital map with the observation data to generate an augmented digital map; and controlling the autonomous vehicle using the augmented digital map.

[0013] According to another aspect of the present invention, an autonomous vehicle control system includes one or more processors and a memory storing instructions that, when executed by the one or more processors, cause the autonomous vehicle control system to receive a digital map of a portion of an environment in which an autonomous vehicle is operating, the digital map defining a plurality of static elements within the portion of the environment, determine observation data related to one or more observations collected from the environment based on sensor data collected using one or more sensors of the autonomous vehicle, augment the digital map with the observation data to generate an augmented digital map, control the autonomous vehicle using the augmented digital map, and forward the observation data to a remote real-time mapping system for use in controlling one or more other autonomous vehicles operating within the environment.

[0014] Also, in some embodiments, the one or more processors are configured to control the autonomous vehicle using the augmented digital map by using the augmented digital map to determine a trajectory of the autonomous vehicle and controlling the autonomous vehicle along the trajectory.

[0015] In some embodiments, the observation data is first observation data, and the one or more processors are further configured to receive second observation data related to one or more observations collected from the environment from the remote real-time map system and augment the digital map with the second observation data, such that the augmented digital map is augmented using the first and second observation data. In some embodiments, the observation data defines construction elements detected in the environment using sensors of the autonomous vehicle. In some embodiments, the observation data defines occluded areas in the environment that restrict vehicle movement within the environment.

[0016] Also, in some embodiments, the observational data defines one or more vehicle paths, each vehicle path being related to a detected path of another vehicle or the path of the autonomous vehicle while traveling through the environment, and the one or more processors are configured to transfer the observational data to the remote real-time map system by transferring the one or more vehicle paths to the remote real-time map system for use in controlling the one or more other autonomous vehicles operating within the environment.

[0017] Also, in some embodiments, the observational data defines a sub-optimal action to be taken by the autonomous vehicle while navigating the environment, and the one or more processors are configured to forward the observational data to the remote real-time mapping system for use in controlling the one or more other autonomous vehicles operating within the environment by forwarding the sub-optimal action to the remote real-time mapping system.

[0018] According to another aspect of the invention, a method of operating an autonomous vehicle using an autonomous vehicle control system may include receiving a digital map of a portion of an environment in which the autonomous vehicle is operating, the digital map defining a plurality of static elements within the portion of the environment; determining observation data related to one or more observations collected from the environment based on sensor data collected using one or more sensors of the autonomous vehicle; augmenting the digital map with the observation data to generate an augmented digital map; controlling the autonomous vehicle using the augmented digital map; and forwarding the observation data to a remote real-time mapping system for use in controlling one or more other autonomous vehicles operating in the environment.

[0019] According to another aspect of the invention, a method of operating an autonomous vehicle may include receiving observational data relating to one or more observations collected from the environment from a remote real-time map system, the observational data defining a plurality of vehicle paths, each vehicle path relating to detected paths of other vehicles while navigating the environment; planning a movement using the received observational data to generate a movement path for the autonomous vehicle based at least in part on the plurality of vehicle paths; and controlling the autonomous vehicle using the movement path.

[0020] Also, in some embodiments, a first vehicle path of the plurality of vehicle paths is associated with a detected path of a first vehicle detected using one or more sensors of a second vehicle operating in the environment. Also, in some embodiments, the first vehicle is a non-autonomous vehicle. Also, in some embodiments, a plurality of vehicle paths of the plurality of vehicle paths is associated with non-autonomous vehicles. In some embodiments, the observational data further includes sub-optimal actions performed by other autonomous vehicles while navigating the environment, and planning movement using the received observational data to generate the movement path for the autonomous vehicle is further based at least in part on the sub-optimal actions.

[0021] According to another aspect of the invention, a method may include conducting a teleassist session with a first autonomous vehicle operating within an environment, the teleassist session including exchanging situational data and teleassist driver input between the first autonomous vehicle and a remote teleassist system; generating a location-based teleassist trigger in association with the teleassist session; and forwarding the location-based teleassist trigger to a remote real-time map system, the remote real-time map system forwarding the location-based teleassist trigger to a second autonomous vehicle.

[0022] Also, in some embodiments, the location-based teleassist trigger includes a session proposal that selectively proposes automatic initiation of a teleassist session with the second autonomous vehicle when the second autonomous vehicle satisfies location-based criteria associated with the location-based teleassist trigger. In some embodiments, the location-based teleassist trigger includes a proposed action to be performed by the second autonomous vehicle when the second autonomous vehicle satisfies location-based criteria associated with the location-based teleassist trigger.

[0023] Some embodiments may also include, at the first autonomous vehicle, receiving observation data associated with one or more observations collected from the environment from the remote real-time map system and controlling the autonomous vehicle using the received observation data. Some embodiments may further include selectively propagating collected observation data associated with the remote teleassist system to the remote real-time map system in response to teleassist driver input.

[0024] According to another aspect of the invention, a method may include receiving observation data from a remote real-time map system relating to one or more observations collected from an environment in which a first autonomous vehicle operates; controlling the first autonomous vehicle using the observation data; receiving from the remote real-time map system a location-based teleassist trigger generated in connection with a teleassist session conducted between a remote teleassist system and a second autonomous vehicle; and determining activation of the location-based teleassist trigger in response to a determination that the first autonomous vehicle satisfies location-based criteria associated with the location-based teleassist trigger.

[0025] In some embodiments, the teleassist session is a first teleassist session, and the location-based teleassist trigger includes a session proposal that selectively proposes automatic initiation of a second teleassist session, and the method further includes automatically initiating the second teleassist session between the remote teleassist system and the first autonomous vehicle when the first autonomous vehicle satisfies the location-based criteria associated with the location-based teleassist trigger. Also, in some embodiments, the location-based teleassist trigger includes a proposed action to be performed by the first autonomous vehicle, and the method further includes controlling the first autonomous vehicle based at least in part on the proposed action.

[0026] According to another aspect of the present invention, a real-time map system includes one or more processors and a memory storing instructions that, when executed by the one or more processors, cause the real-time map system to maintain observation data relating to a plurality of observations collected by a plurality of autonomous vehicles operating within an environment, maintain a location-based teleassist trigger generated in association with a teleassist session between a remote teleassist system and a first autonomous vehicle of the plurality of autonomous vehicles, and transfer a portion of the observation data and the location-based teleassist trigger to the second autonomous vehicle of the plurality of autonomous vehicles for use in controlling the second autonomous vehicle.

[0027] Some embodiments may also include an autonomous vehicle and / or a system remotely located from the autonomous vehicle, including one or more processors configured to perform the various methods described above. Some embodiments may also include an autonomous vehicle control system including one or more processors, a computer-readable storage medium, and computer instructions resident on the computer-readable storage medium and executable by the one or more processors to perform the various methods described above. Other embodiments may include a non-transitory computer-readable storage medium storing computer instructions executable by one or more processors to perform the various methods described above.

[0028] It should be noted that all combinations of the foregoing concepts and additional concepts described in more detail herein are considered to be part of the subject matter disclosed herein, for example, all combinations of claimed subject matter listed at the end of this disclosure are considered to be part of the subject matter disclosed herein. [Brief explanation of the drawings]

[0029] [Figure 1] 1 illustrates an exemplary hardware and software environment for an autonomous vehicle. [Figure 2]FIG. 1 is a block diagram illustrating an exemplary teleassisted assistance system according to some embodiments. [Figure 3] FIG. 3 is a block diagram further illustrating the teleassist control module referred to in FIG. 2. [Figure 4] FIG. 1 is a block diagram illustrating an autonomous vehicle's interaction with an exemplary real-time map and teleassist system, according to some embodiments. [Figure 5] FIG. 1 is a block diagram of a real-time map and teleassist assistance system for controlling an autonomous vehicle according to some embodiments. [Figure 6] FIG. 10 is a block diagram of another real-time map and teleassist assistance system for controlling an autonomous vehicle in accordance with some embodiments. [Figure 7] FIG. 1 is a block diagram illustrating an exemplary real-time map and teleassist assisted autonomous vehicle control system in accordance with some embodiments. [Figure 8] FIG. 8 is a block diagram illustrating a real-time map system according to some embodiments that can interact with the autonomous vehicle control system of FIG. [Figure 9] 9 is a flowchart illustrating an exemplary operational sequence for controlling an autonomous vehicle using the autonomous vehicle control system and real-time map system of FIGS. 7-8. [Figure 10] 9 is a flowchart illustrating an exemplary operational sequence for generating and storing a location-based teleassist trigger in the real-time map system of FIG. 8. [Figure 11] 9 is a flowchart illustrating an exemplary operational sequence for activating a location-based teleassist trigger using the autonomous vehicle control system and real-time map system of FIGS. 7-8. [Figure 12] 9 is a flowchart illustrating an exemplary operational sequence for controlling an autonomous vehicle based on other vehicle paths and / or suboptimal behavior using the autonomous vehicle control system and real-time map system of FIGS. 7-8. [Figure 13]FIG. 9 is a block diagram illustrating an example fusion of real-time map data and a road area layout using the autonomous vehicle control system and real-time map system of FIGS. 7-8 to generate an augmented road area layout. [Figure 14] FIG. 9 is a block diagram illustrating an example fusion of real-time map data and a road area layout using the autonomous vehicle control system and real-time map system of FIGS. 7-8 to generate an augmented road area layout. DETAILED DESCRIPTION OF THE INVENTION

[0030] The various embodiments described below apply in part generally to real-time map systems used with autonomous vehicles to supplement digital maps. However, before describing these embodiments, an exemplary hardware and software environment in which the various techniques disclosed herein may be implemented is described. Hardware and Software Environment

[0031] Referring to the drawings, wherein like numerals indicate like parts throughout the various views, FIG. 1 illustrates an exemplary autonomous vehicle 100 in which various techniques disclosed herein may be implemented. For example, vehicle 100 is shown traveling along a roadway 101. Vehicle 100 may include a powertrain 102 including a prime mover 104, which may be driven by an energy source 106 to power a drivetrain 108, and a control system 110 including directional control 112, powertrain control 114, and brake control 116. It will be understood that vehicle 100 may be implemented as various types of vehicles capable of transporting people and / or cargo and capable of traveling on land, sea, air, underground, undersea, and / or in space, and that the components 102-116 described above may vary significantly depending on the type of vehicle in which these components are used.

[0032] The embodiments described below will focus on wheeled land vehicles, such as cars, vans, trucks, buses, etc. In such embodiments, prime mover 104 may include one or more electric motors and / or internal combustion engines (or others), and energy source 106 may include a fuel system (e.g., providing gasoline, diesel, hydrogen, etc.), a battery system, solar panels or other renewable energy sources, a fuel cell system, etc. Drivetrain 108 may include wheels and / or tires, as well as a transmission and / or other mechanical drive components suitable for converting the power output of prime mover 104 into vehicle motion, one or more brakes configured to controllably stop or slow the vehicle, and direction or steering components suitable for controlling the vehicle's trajectory (e.g., a rack-and-pinion steering linkage that pivots one or more wheels of vehicle 100 about a substantially vertical axis to change the angle of the wheel's plane of rotation relative to the vehicle's longitudinal axis). In some embodiments, a combination of powertrain and energy source can be used, for example, in the case of an electric / gas hybrid vehicle, in some cases multiple electric motors (e.g., dedicated to individual wheels or axles) can be used as prime movers. In the case of hydrogen fuel cell embodiments, the prime mover can include one or more electric motors, and the energy source can include a fuel cell system powered by hydrogen fuel.

[0033] Directional control 112 may include one or more actuators and / or sensors to control and receive feedback from directional or steering components to enable the vehicle to follow a desired trajectory. Powertrain control 114 may be configured to control the output of powertrain 102, for example, by controlling the output power of prime mover 104 or by controlling the transmission gears of drivetrain 108 to control the speed and / or direction of the vehicle. Brake control 116 may be configured to control one or more brakes, for example, disc brakes or drum brakes coupled to the wheels of the vehicle, to slow or stop vehicle 100.

[0034] Those of ordinary skill in the art with reference to this disclosure will recognize that other vehicle types, including, but not limited to, off-road vehicles, all-terrain or tracked vehicles, construction equipment, etc., necessarily use other powertrains, drivetrains, energy sources, directional control, powertrain control, and braking control. Also, in some embodiments, some components may be combined, for example, where vehicle directional control is primarily handled by adjusting the output of one or more prime movers. Accordingly, the present invention is not limited to the specific application of the technology described herein to wheeled autonomous land vehicles.

[0035] In the illustrated embodiment, autonomous control for vehicle 100 (which may include various levels of autonomy and selective autonomous functionality) is primarily implemented in a main vehicle control system 120, which may include one or more processors 122 and one or more memories 124. Each processor 122 is configured to execute program code instructions 126 stored in memory 124.

[0036] The primary sensor system 130 may include various sensors that collect information about the vehicle's surrounding environment for use in controlling the vehicle's operation. Satellite navigation (SATNAV) sensors 132, such as sensors compatible with various satellite navigation systems like GPS, GLONASS, Galileo, and Compass, can be used to determine the vehicle's terrestrial position using satellite signals. Radio detection and ranging (RADAR) and light detection and ranging (LIDAR) sensors 134, 136 and a digital camera 138 (which may include various types of imaging devices capable of capturing still and / or video images) can be used to detect stationary and moving objects in proximity to the vehicle. The inertial measurement unit (IMU) 140 may include multiple gyroscopes and accelerometers that can detect the vehicle's linear and rotational motion in three directions, and one or more wheel encoders 142 can be used to monitor the rotation of one or more wheels of the vehicle 100.

[0037] The outputs of sensors 132-142 may be provided to a set of primary control subsystems 150, which includes a position estimation subsystem 152, a planning subsystem 154, a perception subsystem 156, and a control subsystem 158. The primary role of position estimation subsystem 152 is to accurately determine the vehicle's 100 position and orientation (sometimes referred to as "pose," which may in some cases include one or more of velocity and / or acceleration) within a particular coordinate system, typically the surrounding environment. The primary role of planning subsystem 154 is to plan a path of motion over a period of time given the vehicle's 100 desired destination and stationary and moving objects within the environment, while the primary role of perception subsystem 156 is to detect, track, and / or identify elements within the vehicle's 100 environment. The primary role of control subsystem 158 is to generate appropriate control signals to control various controllers within control system 110 to implement the vehicle's planned path. Any or all of the position estimation subsystem 152, planning subsystem 154, perception subsystem 156, and control subsystem 158 may have associated data generated and / or utilized in connection with the operation and, in some embodiments, may have associated data that can be communicated to the teleassist system.

[0038] The illustrated embodiment also provides a map or map subsystem 160 that describes elements in the environment and the relationships between them. The map subsystem 160 is accessed by each of the localization, planning, and recognition subsystems 152-156 to obtain various information about the environment for use in performing their respective functions. The map subsystem 160 can be used to provide map data to an autonomous vehicle control system, which can be used for various purposes in an autonomous vehicle, such as localization, planning, and recognition. The map data can be used, for example, to layout or position elements within a particular geographic area. Elements include elements that represent real-world objects, such as roads and boundaries (e.g., barriers, lane dividers, medians, etc.), buildings, and traffic devices (e.g., traffic or road signs, traffic signals, etc.), as well as elements that are logical or virtual in nature, such as elements that represent valid paths a vehicle can travel in the environment, "virtual" boundaries such as lane markings, or logical collections or sets of other elements. Map data may include data that characterizes or describes elements in an environment (e.g., data describing the geometry, size, shape, etc. of an object) or data that describes the type, function, behavior, purpose, etc. of elements in an environment (e.g., speed limits, lane restrictions, operation or logic of traffic devices, etc.). In some embodiments, map subsystem 160 may provide map data in a format in which the positions of at least some of the elements within a geographic region are defined primarily based on relative positions between elements rather than absolute positions within a geographic coordinate system. However, it will be appreciated that in other embodiments, other maps or map systems suitable for maintaining map data for use by autonomous vehicles can be used, including systems based on absolute positions. It will also be appreciated that in some embodiments, at least a portion of the map data generated and / or used by map subsystem 160 may be distributed to a teleassist system.

[0039] It will be appreciated that the collection of components of the main vehicle control system 120 shown in FIG. 1 is merely exemplary in nature. In some embodiments, individual sensors may be omitted, multiple sensors of the type shown in FIG. 1 may be used for redundancy and / or to cover different areas around the vehicle, and other types of sensors may be used. Similarly, in other embodiments, other types and / or combinations of control subsystems may be used. Also, while the subsystems 152-160 are shown as separate from the processor 122 and memory 124, it will be appreciated that in some embodiments, some or all of the subsystem functionality 152-160 may be implemented as program code instructions 126 resident in one or more memories 124 and executed by one or more processors 122, and that these subsystems 152-160 may, in some cases, be implemented using the same processor and / or memory. Subsystems according to some embodiments may be implemented in part using various dedicated circuit logic, various processors, various field programmable gate arrays (FPGAs), various application-specific integrated circuits (ASICs), various real-time controllers, etc., and, as previously discussed, multiple subsystems may utilize common circuitry, processors, sensors, and / or other components. Additionally, the various components of the main vehicle control system 120 may be networked in a variety of ways.

[0040] In some embodiments, vehicle 100 may include an auxiliary vehicle control system 170, which may be used as a redundant or backup control system for vehicle 100. In some embodiments, auxiliary vehicle control system 170 may be capable of fully operating autonomous vehicle 100 in the event of an abnormal situation occurring in primary vehicle control system 120. In other embodiments, auxiliary vehicle control system 170 may have only limited functionality, for example, performing a controlled stop of vehicle 100 in response to an abnormal situation detected by primary vehicle control system 120. In yet other embodiments, auxiliary vehicle control system 170 may be omitted.

[0041] In general, many different architectures can be used to implement the various components shown in FIG. 1 , including various combinations of software, hardware, circuit logic, sensors, networks, etc. Each processor can be implemented, for example, by a microprocessor. Each memory can represent a random access memory (RAM) device constituting primary storage, as well as secondary memory layers such as cache memory, non-volatile or backup memory (e.g., programmable or flash memory), and read-only memory. Each memory can also include memory storage devices physically located elsewhere within vehicle 100, such as cache memory within a processor, or storage capacity stored in a mass storage device or other computer or controller and used as virtual memory. One or more processors shown in FIG. 1 or entirely separate processors can be used to implement additional functions beyond those intended for autonomous control of vehicle 100, such as controlling an entertainment system and operating vehicle doors, lights, and convenience features.

[0042] For additional storage, vehicle 100 may also include one or more mass storage devices, such as a floppy disk or other removable disk drive, a hard disk drive, a direct access storage device (DASD), an optical drive (e.g., a CD drive, a DVD drive, etc.), a solid-state storage drive (SSD), a network-attached storage (NAS), a storage area network, a tape drive, etc. Vehicle 100 may also include a user interface 172 that enables vehicle 100 to receive multiple user or driver inputs and generate outputs for the user or driver, examples of which include one or more displays, a touchscreen, a voice and / or gesture interface, buttons and other tactile controls, etc. Alternatively, user input may be received from a remote driver via another computer or electronic device, such as via an app on a mobile device or via a web interface.

[0043] Vehicle 100 may also include one or more network interfaces, e.g., network interface 174, that enable it to communicate with one or more networks 176 (e.g., a LAN, a WAN, a wireless network, and / or the Internet) to communicate information with other vehicles, computers, and / or electronic devices. The network interfaces may include central services, e.g., cloud services, from which vehicle 100 receives environmental and other data used for autonomous control. In the illustrated embodiment, for example, vehicle 100 may communicate with various cloud-based remote vehicle services 178, including a map or map service or system 180, a teleassist service or system 182, and a real-time map service or system 184, to perform at least the various functions described herein. Map or map service or system 180 may be used, for example, to maintain a global repository describing one or more geographic regions of the world, distribute portions of the global repository to one or more autonomous vehicles, and update or otherwise manage the global repository according to information received from one or more autonomous vehicles. Teleassist service or system 182 may be used to provide teleassist assistance to vehicle 100, for example, by communicating with teleassist subsystem 186 located within main vehicle control system 120, as described in more detail below. Real-time map service or system 184 may be used to propagate various observations collected by one or more autonomous vehicles, effectively supplementing the global repository maintained by map or map service or system 180. The terms "service" and "system" are generally used interchangeably herein and generally refer to computer functionality that may receive data from and provide data to an autonomous vehicle. In many cases, these services or systems may be considered remote services or systems insofar as they are external to and communicate with the autonomous vehicle.

[0044] 1 and the various additional controllers and subsystems disclosed herein generally operate under the control of an operating system and execute or rely on various computer software applications, components, programs, objects, modules, data structures, etc., which are described in more detail below. The various applications, components, programs, objects, modules, etc. may also execute on one or more processors of other computers connected to vehicle 100 over a network, for example, in a distributed, cloud-based, or client-server computing environment, such that the processing required to perform the functions of a computer program may be allocated to multiple computers and / or services over a network. Also, in some embodiments, data recorded or collected by the vehicle may be manually retrieved and uploaded to other computers or services for analysis.

[0045] In general, the routines executed to implement the various embodiments described herein are referred to herein as “program code,” whether implemented as part of an operating system, a specific application, component, program, object, module, or sequence of instructions, or as a subset thereof. Program code generally comprises one or more instructions that reside at various times in various memory and storage devices, which, when read and executed by one or more processors, perform the steps necessary to perform the steps or elements that implement various aspects of the present invention. Also, while the present invention has been described and will be described below in the context of fully functional computers and systems, it will be understood that the various embodiments described herein may be distributed in various forms of program products, and the present invention applies equally regardless of the particular type of computer-readable medium used to actually carry out the distribution. Examples of computer-readable media include tangible, non-transitory media, such as volatile and non-volatile memory devices, floppy and other removable disks, solid-state drives, hard disk drives, magnetic tape, optical disks (e.g., CD-ROMs, DVDs, etc.), and the like.

[0046] Additionally, various program code described below may be identified based on the application for which it is implemented in a particular embodiment. However, it should be noted that any particular program nomenclature below is used merely for convenience, and the present invention should not be limited to use only with any particular application identified and / or implied by such nomenclature. Also, given the typically myriad ways in which computer programs can be organized into routines, procedures, methods, modules, objects, etc., and the various ways in which program functionality can be allocated among the various software layers (e.g., operating system, libraries, APIs, applications, applets, etc.) resident within a typical computer, it should be noted that the present invention is not limited to the specific organization and allocation of program functionality described herein.

[0047] Those skilled in the art will appreciate that the exemplary environment illustrated in Figure 1 is not intended to limit the scope of the present invention. Indeed, those skilled in the art will recognize that other alternative hardware and / or software environments may be used without departing from the scope of the present invention. Teleassist System

[0048] When operating autonomous vehicles in the complex dynamic environments in which automobiles regularly operate, they must often handle a variety of combinations of conditions that, while relatively rare, are frequently encountered by many autonomous vehicles over time. Handling these rare conditions efficiently and autonomously can be challenging, and some proposed approaches to resolving these rare conditions include utilizing teleassist or "human-in-the-loop" technology, where a human driver, who may be remotely located, can make decisions and assist with vehicle guidance whenever the vehicle encounters these conditions.

[0049] Some proposed teleassistance methods focus on direct vehicle control by a remote driver, where the driver is provided with sensor data collected by the vehicle and can directly control the vehicle from a remote location. However, it has become clear that direct vehicle control in such situations typically requires a fast, responsive, and reliable network connection between the remote driver and the autonomous vehicle. However, the network connectivity and latency of an autonomous vehicle can vary significantly depending on its location (e.g., urban or rural, highway or local road, etc.) and network congestion. Furthermore, even when provided with sensor data collected by the vehicle, the remote driver may not have full situational awareness because they are not physically present inside the vehicle.

[0050] However, many of the embodiments disclosed herein below may focus on indirect control schemes in which a teleassist service or system may provide suggestions or recommendations to an autonomous vehicle, but these instructions or recommendations must be verified before being acted upon by the autonomous vehicle, thereby effectively decoupling the performance of the vehicle from the performance of the network connecting the teleassist service or system and the autonomous vehicle.

[0051] FIG. 2 , for example, illustrates one embodiment of a teleassist assistance system or service 200 in which an autonomous vehicle 202 interfaces with a remote teleassist system 204 via a network 206. The remote teleassist system 204 may be physically separate from the autonomous vehicle 202 and typically supports interfacing with multiple vehicles, allowing multiple teleassist drivers to simultaneously interact with multiple vehicles. As will become more apparent below, in some embodiments, a teleassist driver may actively and continuously monitor individual vehicles, while in other embodiments, individual teleassist drivers may interact with multiple vehicles at different times, e.g., such that a particular driver may assist multiple vehicles at once. In some embodiments, for example, a teleassist driver may selectively connect to a particular autonomous vehicle as needed, e.g., in response to vehicle-generated requests whenever a particular condition occurs (e.g., various special situations where teleassist assistance may be useful). In some embodiments, a driver pool supports a pool of autonomous vehicles, and the teleassist system may initiate teleassist sessions as needed based on requests initiated by the autonomous vehicles, the teleassist system, or both.

[0052] In some embodiments, teleassist assistance can be implemented using a teleassist control module 208 and a teleassist camera module 210 in the autonomous vehicle 202 that communicate with a teleassist base module 212 in the teleassist system 204. The vehicle's modules 208-210 can be connected to the network 206 via a modem 214, and the teleassist system 204's module 212 can be connected to the network 206 via a modem aggregator unit 216 that can communicate with multiple modems 214 of multiple autonomous vehicles 202. The network 206 can be implemented in part using a wireless network, such as a 4G, LTE, or 5G network, a satellite network, or a combination thereof, although the invention is not limited thereto.

[0053] Teleassist control module 208, in some embodiments, may be provided within main computing system 218 of vehicle 202 and may interface with vehicle's autonomous system 220 and platform 222 to collect and stream data from the main computing system to teleassist system 204, and to receive and process driver input received from teleassist system 204. In some embodiments, main computing system 218 may be implemented in a manner similar to main vehicle control system 120 shown in FIG. 1 , with autonomous system 220 representing high-level autonomous control subsystems such as localization, planning, perception, etc., and platform 222 representing low-level vehicle controls such as those provided by control subsystem 158. However, it will be appreciated that in other embodiments, teleassist control module 208 may interface with any autonomous or control-related aspect of vehicle 202.

[0054] In some embodiments, the teleassist camera module 210 may be located within a camera system 224 that manages the onboard cameras of the vehicle 202, and in some embodiments, the module 210 can stream camera feed data collected from the onboard cameras to the teleassist system 204 for viewing by the driver during a teleassist session. In some embodiments, the module 210 can dynamically change the data streamed from the onboard cameras. For example, the module 210 can change the priority, quality, and / or resolution of each camera feed.

[0055] 2, modules 208-210 are implemented independently, but in other embodiments, the functions assigned to each module may vary, functions may be combined into a single module, or functions may be split into two or more modules. Thus, the present invention is not limited to the specific architecture shown in FIG. 2.

[0056] Teleassist base module 212 communicates with modules 208-210 during a teleassist session with vehicle 202 and can additionally manage multiple sessions with multiple vehicles and multiple drivers. In some embodiments, module 212 can also manage the scheduling, starting, and ending of sessions.

[0057] Teleassist driver user interface 226 is connected to module 212 and provides a user interface through which a driver, e.g., a human driver, can communicate with vehicle 202 during a session. The user interface can be implemented in a number of suitable ways, including using text, graphics, video, audio, virtual or augmented reality, keyboard input, mouse input, touch input, voice input, gesture input, etc. In some embodiments, dedicated or customized controls and / or indicators can be used. Also, in some embodiments, an application that can run on a desktop computer, laptop computer, mobile device, etc. can be used to interact with the driver, while in other embodiments, a web-based or remote interface can be used. In one exemplary embodiment, described in detail below, for example, interface 226 can be a web-based interface that interacts with the driver via a touchscreen display.

[0058] Teleassist system 204 may include one or more autonomous components 228 interfaced with module 212. Autonomous components 228 may include components accessible to the vehicle for use in connection with the vehicle's primary controls (e.g., components 240, 242, 244, described below in connection with FIG. 3) to replicate the functionality of similar components within vehicle 202 and / or to replicate functionality. For example, in some embodiments, module 212 may have access to the same map data used by each vehicle, e.g., provided by the aforementioned map systems, and similar layout functions used by each vehicle to layout the map data in proximity to the vehicle. This allows module 212 to reduce the amount of data transmitted by the vehicle to reconstruct the environment around the vehicle by effectively reconstructing a digital map for nearby static objects around the vehicle without receiving the entire digital map from the vehicle itself. In some embodiments, the vehicle may provide data regarding the vehicle's current pose and any dynamic entities detected by the recognition system (e.g., other vehicles, pedestrians, or other actors or objects detected in the environment but not displayed in the map data), and from this limited amount of data, a graphical representation of the nearby area around the vehicle may be displayed to the teleassist driver. In some embodiments, the autonomous component may also be capable of local evaluation of how the vehicle can react to specific instructions of the teleassist system by replicating functionality implemented in the vehicle 202, and in some embodiments, the autonomous component may have functionality similar to that implemented in the vehicle 202, but may have greater capabilities and / or access to more computing resources than are available in the vehicle.

[0059] Additionally, because the teleassist system may have autonomy, the teleassist system acts similarly to a teleassist driver with whom the autonomous vehicle interacts during a teleassist session. In this case, the teleassist system may assess the autonomous vehicle's current situation and send commands, requests, instructions, suggestions, etc. to resolve the condition that triggered the teleassist session. In some embodiments, for example, the teleassist system has access to computing power beyond that which may be practically provided onboard the autonomous vehicle, and thus the teleassist system may perform computationally complex assessments to assist the autonomous vehicle.

[0060] The teleassist system 204 may include an operational / fleet interface 230 to facilitate communication with other services and / or systems assisting the autonomous vehicle. For example, in some embodiments, it may be desirable to provide the ability to request roadside assistance or recovery for the autonomous vehicle, or log data used to diagnose vehicle problems. It may also be desirable to distribute data collected during a teleoperation session (e.g., data related to lane closures, detected construction, or accidents) to other vehicles in the fleet. Data received and / or generated by the teleassist system may also be used to further train various components of the autonomous vehicle. For example, it may be used to improve detector performance and reduce the occurrence of false alarms, or to improve scenario selection and other decisions of the autonomous vehicle in response to specific sensor inputs. In other embodiments, other external services and / or systems may interface with the teleassist system, as would be apparent to one of ordinary skill in the art.

[0061] 3, an exemplary embodiment of teleassist control module 208 and the various interfaces it supports will be described in further detail. In some embodiments, for example, module 208 may include an interface to teleassist system 204 (via modem 212), an interface to platform 222, and an interface to each of perception component 240, map layout component 242, and planner component 244 of autonomous driving component 220, where these interfaces are, for example, application programming interfaces (APIs).

[0062] To interface with the teleassist system 204, the teleassist control module 208 may be configured to forward autonomy data (e.g., map data, perception data, route data, planning data), sensor data, telemetry data, etc. to the teleassist system 204 to provide the teleassist system with the current state of the vehicle. The teleassist control module 208 may be configured to forward teleassist requests to the teleassist system 204 and receive various teleassist commands from the teleassist system. The teleassist control module 208 may also be configured to receive visualization requests and forward the requested visualization requests to the teleassist system 204.

[0063] To interface with the platform 222, the teleassist control module 208 may be configured to receive vehicle status information (e.g., various types of diagnostic and / or sensor data) from the platform and to issue various lower-level commands to the platform, such as, for example, sounding the horn, activating or deactivating emergency lights, changing gears, retiring the vehicle, initiating a controlled stop, etc.

[0064] To interface with the perception component 240, the teleassist control module 208 may be configured to receive from the perception component information about actors and / or their trajectories detected in the environment, detection results from various detectors 246 implemented in the perception component, and other perception-related data, all of which may be transmitted by the module 208 to the teleassist system as autonomous data.

[0065] By interfacing with the map layout component 242, the teleassist control module 208 may receive, for example, regional map data, route data, and other map-related data from the map layout component 242. In some cases, the module 208 can forward map patches to the map layout component 242, for example, to generate lane closures, traffic device overrides, new destinations, virtual route suggestions, etc., or to remove previously generated map patches that were applied to the regional map stored in the component 242, for example, when a previous lane closure was removed. In some cases, the map layout component 242 can send map and route updates to the planner component 244 to update scenarios that the planner component is considering during operation of the autonomous vehicle.

[0066] By interfacing with the planner component 244, the teleassist control module 208 may receive, for example, generated plans, actor attributes, alternative scenarios, and other plan-related data. The teleassist control module 208 may send various teleassist commands to the planner component 244 and may receive feedback from the planner component 244 about teleassist requests and / or teleassist commands.

[0067] Those of ordinary skill in the art with reference to this disclosure will recognize other features and variations, and therefore the present invention is not limited to the specific teleassist system embodiments described herein. Remote real-time mapping system for autonomous vehicles

[0068] It will be appreciated that an autonomous vehicle preferably recognizes objects and elements in its surrounding environment well in advance, allowing it sufficient time to respond to them. This is particularly important in the case of stationary objects in the lane in which the autonomous vehicle is traveling, such as construction elements (e.g., cones and barrels) that define a construction zone and / or block a portion of the road. While the autonomous vehicle's sensors and recognition systems can detect these objects and elements, the detection range is often such that some objects may not be detected early enough to adequately search the construction zone. In some cases, road signs informing drivers of planned construction zones may be detected, providing early notification of a potential lane closure. However, even if road signs are present, the vehicle's sensors may be blocked by surrounding vehicles, preventing the autonomous vehicle from providing sufficient advance notice. As an example, an autonomous vehicle traveling alongside relatively tall commercial trucks or vans on a roadway may have a limited view of road signs or the road ahead. Additionally, while a teleassist system can potentially be useful for an autonomous vehicle to search for a construction zone (e.g., because road signs are obscured by other vehicles), if the autonomous vehicle does not have sufficient advance knowledge of the existence of the construction zone, there may not be enough time to set up a teleassist session before encountering the construction zone.

[0069] For example, offline map data maintained in a global repository and used to represent static objects or elements in an environment may contain construction-related information about the environment. For long-term construction projects in particular, lane closures or changes may persist for weeks or months and, therefore, be reflected in the offline map data used by autonomous vehicles in that environment. However, because the update cycle for such map data is a heavy-duty offline process that typically requires several days, temporary or long-term construction zone changes (e.g., lane closure changes) may not be immediately reflected in the offline map data used by autonomous vehicles. This discrepancy between the offline map data and the autonomous vehicle's real-time perception can pose challenges to exploration and motion planning.

[0070] However, in some embodiments of the present invention, a real-time mapping service or system may be used to utilize different observations collected by different autonomous vehicles operating in the same environment to supplement the offline map data used by the autonomous vehicles. In particular, in some embodiments, a real-time mapping system may receive observational data from one or more autonomous vehicles operating in the environment, maintain the observational data in a data store, and then forward the observational data to one or more autonomous vehicles operating in the environment, such that observational data generated by some autonomous vehicles in the environment can be used by other autonomous vehicles to control those vehicles.

[0071] To this end, observational data may include various types of data related to one or more observations collected in the environment and usable by the autonomous vehicle control system in connection with controlling the autonomous vehicle. In some cases, observational data may relate to cognitive observations, which may include observations detected by one or more sensors of the autonomous vehicle and / or derived from data detected by such sensors. Such observations may include, for example, sensor data such as image data, LIDAR data, telemetry data, etc., and the trajectories, paths, and / or positions of objects or elements detected in the environment, where the detected objects or elements may be, for example, static or fixed objects or elements, dynamic or moving objects or elements, physical objects or elements, logical objects or elements, etc. In some embodiments, observational data based on cognitive observations may include, for example, the trajectories of signs and / or construction elements (e.g., cones, barrels, Jersey Barriers, etc.) detected in the environment, the trajectories of other vehicles, pedestrians, and / or other moving objects in the environment, physical and / or logical boundaries of roads (e.g., those defining lanes, shoulders, etc.), occluded areas representing undriveable areas, etc. In some embodiments, the occlusion area can be used to define an area of ​​a digital map where traffic is blocked, and in some cases, for example, to represent multiple related construction elements detected in the environment, thereby avoiding the need to track all the construction elements individually. The observation data can also include data related to the accident scene, such as police cars, cones, or other construction elements temporarily placed on the road to block off the accident scene, the occlusion area defined by these, the movement paths of other vehicles traveling around the accident scene, etc.

[0072] The observational data may relate to operational observations based on decisions and / or actions taken by the autonomous vehicle while it is operating within an environment and / or performance evaluations of such decisions and / or actions (e.g., whether the decisions were determined to be good or bad decisions, whether the determined motion plan provided for smooth and efficient operation of the autonomous vehicle, etc.). In some embodiments, for example, the observational data based on operational observations may include the determined motion plan and its relative performance, and in some cases, optimal and / or sub-optimal operations may be identified. The optimal operation may include, for example, an operation or maneuver determined to provide desired performance of the autonomous vehicle when responding to a particular situation. The sub-optimal operation may include, for example, an operation or maneuver performed by the autonomous vehicle that is determined to provide sub-optimal performance when the motion plan requires abrupt corrections in direction and / or speed, requires the autonomous vehicle to stop or pull over to the side of the road, or applies adverse forces to the autonomous vehicle and passengers (e.g., going over bumps too quickly, turning sharply, slowing down sharply, etc.).

[0073] Thus, a real-time map system or service can be considered to provide an "online" map system that provides relatively up-to-date observational data collected by other autonomous vehicles, often in real time or near real time. This is contrasted with "offline" map systems, such as map systems that generally provide relatively stable representations of static objects or elements (including physical objects or elements, such as road boundaries and buildings, and logical objects, such as lanes, intersections, and traffic signals). In the illustrated embodiment, the real-time map system can be distinguished from offline map systems based on the recency of the information (e.g., minutes or hours versus days or weeks), the method of propagation to the autonomous vehicles (e.g., current location of the autonomous vehicle and requested on-demand by the autonomous vehicle or forced by the real-time map system versus forwarded to the autonomous vehicle fleet through version updates), and / or the degree of validation and quality assurance (e.g., little or no validation and / or quality assurance in the real-time map system versus comprehensive validation and quality assurance before propagating offline map updates). Additionally, in some embodiments, a real-time map system may be distinguished from an offline map system based on the fact that observational data propagated by the real-time map system may be based on observations of individual autonomous vehicles, whereas map data maintained by an offline map system may be based primarily on aggregated observations collected and / or verified by multiple autonomous vehicles. Note that for purposes of this disclosure, unless expressly stated otherwise, the terms "object" and "element" may be considered synonymous with each other.

[0074] In the context of this disclosure, map data (which may be referred to as offline map data) provided by an offline map system, such as a map system, may be used to generate a digital map representing the environment in which the autonomous vehicle is operating. In some embodiments, such a digital map may be referred to as a road area layout. In some embodiments, the offline map data may be provided by a remote offline map system as the autonomous vehicle operates within the environment. In other embodiments, the offline map data is stored in the autonomous vehicle and available for use locally while operating, and updates to the offline map data are provided periodically (e.g., through version updates distributed to the fleet of vehicles).

[0075] Additionally, the observational data provided by the real-time map system can be used to supplement the offline map data used to generate the digital map, effectively "merging" the observational data with the offline map data, to generate an augmented digital map. For example, the observational data can be fused by supplementing the offline map data (e.g., adding blocked areas, boundaries, construction elements, signs, etc.) and / or replacing or overwriting the offline map data (e.g., closing off normally drivable areas or opening previously undrivable areas). In some embodiments, if the digital map is implemented with a road area layout, the observational data can be used to generate the augmented road area layout.

[0076] Additionally, as will be appreciated from the following description, such augmentation of the digital map can be accomplished through a map fusion or similar functional component of the autonomous vehicle control system that fuses additional information into the digital map or augmented digital map. The additional information can include, for example, cognitive observations generated from the autonomous vehicle's sensor data, such as detected signs, construction elements, the path of other objects and elements detected in the environment, and various lane boundaries detected in the environment. In some embodiments, the map fusion or similar functional component can also fuse additional information received from a remote teleassist system, such as a suggested route, blocked areas, etc.

[0077] While augmented digital maps may be useful in other tasks related to operating autonomous vehicles, the remainder of this disclosure focuses primarily on using augmented digital maps to perform motion planning for autonomous vehicles. Motion planning is considered to include functions and algorithms suitable for generating a motion path for an autonomous vehicle in a relatively short period of time (while path planning focuses on traversing a route to a particular geographic destination, rather than on the actual path the autonomous vehicle will take within roads while traveling along the path).

[0078] In some embodiments, for example, the observational data may include the paths of one or more vehicles (including autonomous and / or non-autonomous vehicles) as they pass through an area, allowing a motion planner to select a motion path for the autonomous vehicle based on these other vehicle paths. Thus, for example, if multiple non-autonomous vehicles follow a particular path through a complex construction zone (e.g., changing lanes or changing to the other side of a divided road), the motion planner may utilize these vehicle paths to generate a similar motion path for the autonomous vehicle. Similarly, for suboptimal behaviors indicated in the observational data, the motion planner may use these behaviors to reject potential motion paths that may induce such suboptimal behavior. Thus, if another autonomous vehicle follows a particular motion path through a complex construction zone that causes the autonomous vehicle to suddenly modify its direction and / or speed, that motion path may be provided to the other autonomous vehicles, which may reject a similar motion path generated by the motion plan.

[0079] Map fusion or similar functional components may also be used to provide observational data collected by an autonomous vehicle to a real-time map system, thereby propagating it to other autonomous vehicles operating in the environment.

[0080] As will be described in more detail, in some embodiments, a real-time map service can be used to propagate teleassist-related information to autonomous vehicles operating in a region. In some embodiments, such information can include a location-based teleassist trigger, which can include teleassist information. The teleassist information can be related to certain location-based criteria that, when met, can be used to automatically trigger a teleassist session and / or provide teleassist-generated guidance to the autonomous vehicle. In some embodiments, for example, a location-based teleassist trigger can include a session suggestion that recommends initiating a teleassist session when the autonomous vehicle is approaching a predetermined location, allowing the teleassist driver to provide guidance to the autonomous vehicle. Such guidance can provide assistance in areas or scenarios such as construction sites, checkpoints, weigh stations, garage entrances / exits, or other complex areas encountered by other autonomous vehicles where teleassist assistance is known to be useful.

[0081] Additionally, in some embodiments, the location-based teleassist trigger may include a suggested action. A suggested action may be, for example, a route to follow, a lane to drive, or other action that the teleassist driver can suggest when the autonomous vehicle is operating in an area related to the location-based criteria. In some embodiments, the suggested action may be displayed within the teleassist session. In some cases, both the session suggestion and the suggested action may be associated with the location-based teleassist trigger, while in other cases, the suggested action may be used to potentially avoid the need to initiate a teleassist session. This allows the autonomous vehicle to effectively traverse a potential problem area without the assistance of the teleassist driver, while still using suggestions that the teleassist driver provided when previously assisting other autonomous vehicles that traversed the area.

[0082] In some embodiments, a location-based teleassist trigger may be related to observational data collected by other autonomous vehicles in association with the other autonomous vehicles and the teleassist session that has been conducted, and this observational data may be used by the other autonomous vehicles. For example, the vehicle path of an autonomous vehicle participating in a teleassist session and / or the vehicle paths of other vehicles detected by the autonomous vehicle may be used in association with the location-based teleassist trigger. Similarly, best and / or suboptimal actions performed by an autonomous vehicle during a teleassist session may be propagated to other autonomous vehicles via the location-based teleassist trigger. Other observational data collected in association with a teleassist session may be propagated to other autonomous vehicles via the location-based teleassist trigger, for example, to assist in the motion control of the autonomous vehicle and / or to assist other teleassist drivers when initiating new teleassist sessions with other autonomous vehicles.

[0083] The location-based criteria used in a location-based teleassist trigger may vary according to various embodiments. In some embodiments, the criteria may be limited to specifying a location that the autonomous vehicle will trigger when within a predetermined distance of a particular location, while in other embodiments, more detailed criteria may be used. For example, defining a geofence that is triggered whenever the autonomous vehicle enters, and / or specifying the path, road, lane, direction, etc. of the autonomous vehicle (e.g., to prevent triggering when the autonomous vehicle is traveling in the opposite direction on a divided road).

[0084] It will also be appreciated that in some embodiments, observation data maintained in the real-time map system may be ad hoc or temporary in nature. For example, a construction zone may be removed or modified (e.g., changing a lane closure from a left lane to a right lane). Thus, in some embodiments, it may be preferable to associate observation data with a validity time or period and automatically remove the observation data from the real-time map system if similar observations have not been collected by other autonomous vehicles over a period of time. It may also be preferable to utilize offline processes for validation and / or quality assurance of the observation data with the assistance of a human driver to remove unreliable observation data and / or propagate reliable observation data to the offline map system for inclusion in the global repository.

[0085] Those of ordinary skill in the art who have the benefit of this disclosure will recognize other variations.

[0086] FIG. 4 illustrates an exemplary system 250 according to some embodiments that utilizes a real-time map system and a teleassist system to assist in the control of an autonomous vehicle. In this embodiment, a map fusion component 252 integrates or fuses various static elements (e.g., elements provided in the form of a road region layout by a road region generator 254) mapped in the autonomous vehicle's surrounding environment by an offline map system with various observations currently detected in the environment by a recognition component 256 to generate a digital map. A motion planner component 258 may use this digital map to generate a route for the autonomous vehicle to follow. The digital map may include static and recognized elements mapped by the offline map system—recognized elements include, for example, detected but unmapped sign tracks, unmapped construction elements, other unmapped objects, and elements detected in the environment—as well as various recognized lane boundaries detected in the environment. Thus, observation data from the recognition component 256 may be used to supplement stored map data with additional elements detected in the environment and / or changes to existing mapped elements to provide a more up-to-date representation of the autonomous vehicle's surrounding environment.

[0087] The real-time map system 258 may also provide observational data related to observations collected by other autonomous vehicles to the map fusion component 252 to augment the digital map with these other observations. Thus, the augmented digital map provided to the motion planner component 258 may include not only offline map data, but also observational data collected in the field by the autonomous vehicle (from the perception component 256) and observational data collected by other autonomous vehicles operating in the same environment (from the real-time map system 258).

[0088] The real-time map system 258 may also provide a location-based teleassist trigger to the map fusion component 252 to trigger a teleassist session with the remote teleassist system 262, for example. For example, in response to the location-based criteria of the location-based teleassist trigger being met, a request for a teleassist session may be sent to the teleassist system 262 via interface 266 (block 264). Feedback generated by the teleassist driver during the teleassist session may be received by the map fusion component 252 via interface 268 and integrated to assist in the operation of the autonomous vehicle (block 270). The interfaces 266, 268 (which in some embodiments may be implemented by the same component) may be used to transfer data, such as situational awareness information, from the map fusion component 252 to the remote teleassist system 262 and to send data, such as driver suggestions, commands, or requests for additional information, from the remote teleassist system 262 to the map fusion component 252, respectively.

[0089] For example, in response to the location-based teleassist trigger triggering a teleassist request in block 264, the remote teleassist system 262 may obtain or generate a teleassist session (block 274) and provide situational awareness information to the teleassist driver (block 276). The situational awareness information or data may include any information or data related to the autonomous vehicle's current context and / or that may be useful to the teleassist driver during the teleassist session. Examples of this information or data include the current route, the paths of other vehicles, map data of the autonomous vehicle's surrounding area, telemetry data, camera feed data, and / or other data that may be useful in assessing the current situation facing the autonomous vehicle. Once the situational data is provided, the remote teleassist system waits for the driver to assess the situation and provide driver input (block 278). This is incorporated into block 270 as described above. After the driver input is provided to the autonomous vehicle, the driver may continue to monitor the autonomous vehicle's operation until the teleassist interaction is complete or may provide additional input (block 280). Once the teleassist interaction is complete, the driver may end the session (block 282).

[0090] Next, FIG. 5 illustrates an exemplary real-time map and teleassist assistance system 300 including an autonomous vehicle 302 in communication with a set of online services 304. A motion planner component 306 receives data, e.g., an augmented road area layout, from a map fusion component 308. Similar to the map fusion component 252 of FIG. 4, the map fusion component 308 receives a digital map, e.g., a road area layout, from a road area generation component 310. The road area generation component 310 may generate the road area layout by accessing an onboard map 312 to retrieve offline map data suitable for generating a digital map of the surrounding environment while the autonomous vehicle 302 is operating. The component 310 may also receive observation data from a recognition component 314, which collects observations using one or more sensors, similar to those described above. In some embodiments, the recognition component may provide tracks of various dynamic objects directly to the motion planner component 306. The map fusion component 308 receives teleassist data, e.g., teleassist driver input, from a teleassist component 316 that communicates with a remote teleassist system 318, and may receive additional observational data from a real-time map system 320. This allows the data collected from components 310, 314, 316, and 320 to be fused into an augmented road terrain layout that the motion planner component 306 uses when generating a motion path for the autonomous vehicle.

[0091] The observational data provided by the real-time map system 320 is stored in a real-time map database or data store 322 and includes observational data collected from one or more other autonomous vehicles 324. The real-time map system 320 generally communicates bidirectionally with both the autonomous vehicle 302 and the other autonomous vehicles 324 to collect observational data for the autonomous vehicles 302, 324 and to forward observational data collected by an autonomous vehicle 302, 324 operating in a particular portion or region of an environment to other autonomous vehicles 302, 324 operating in the same portion or region. The teleassist system 318 may also communicate bidirectionally with the real-time map database 322 via the real-time map system 320, enabling the teleassist driver to store teleassist data, e.g., location-based teleassist triggers, for propagation to the autonomous vehicle via the real-time map system 302, and to retrieve observational data from the real-time map database 322 in connection with conducting a teleassist session with the autonomous vehicle 302. As previously mentioned, in some embodiments, teleassist system 318 (e.g., via teleassist component 316) may communicate bidirectionally with autonomous vehicle 302. This allows the autonomous vehicle to provide situational awareness data to the teleassist driver and the teleassist driver to provide suggested actions to the autonomous vehicle. In some embodiments, there may not be a direct link between teleassist system 318 and real-time map system 320, and therefore communication between these components may be handled via map fusion component 308 and teleassist component 316.

[0092] As previously mentioned, real-time map system 320 and real-time map database 322 are representative online map systems that propagate observation data within a fleet of autonomous vehicles, and are contrasted with map service 326, such as a map system or service that provides offline map data to the fleet. Offline map service 326 may include, for example, offline map database 328, which maintains a global repository of offline map data. A map publishing component 330 may be used to generate versioned updates to the offline map database, and a map distribution component 332 may be used to distribute or propagate the database updates to the fleet of autonomous vehicles. Because the global offline map data store may be large, it is understood that in some embodiments, only a portion of the offline map data applicable to a particular region (e.g., a city, state, county, etc. in which autonomous vehicle 302 operates) may be distributed to the autonomous vehicle and maintained on onboard map 312. Additionally, based on the autonomous vehicle's movement into neighboring regions, additional offline map data may be forwarded to the autonomous vehicle by service 326, such that onboard map 312 may contain sufficient offline map data to navigate the autonomous vehicle at its current location.

[0093] In some embodiments, the real-time map system 320 and the offline map service 326 may be completely independent of one another; in other embodiments, as shown by the system 300′ of FIG. 6 , a map operator component 334 of the offline map service 326′ may communicate with the real-time map system 320′ and incorporate observation data collected by the real-time map system 320′ into the offline map database 328. In this approach, frequently changing observations, such as construction features, may be maintained in both the real-time map database 322 and the offline map database 328 (potentially in a separate layer with lower accuracy requirements than other offline map data), and these observations may be validated (e.g., by the map operator component 334, optionally at the direction of a human driver) before being distributed, and then given priority if they are found to overlap with invalid observations in the real-time map database 322 when loaded onto the autonomous vehicle. This offline map data may also utilize an automatic expiration policy similar to that of the observations in the real-time map database 322, automatically expiring observations that no longer exist in the real world.

[0094] In other embodiments, other methods of partially or fully integrating the real-time map system with the offline map system may be used, as will be understood by those of ordinary skill in the art with reference to this disclosure.

[0095] Referring now to Figure 7, Figure 7 illustrates an exemplary autonomous vehicle 350 including an autonomous vehicle control system 352 that interfaces with both a real-time map system and a teleassist system in association with the autonomously operating autonomous vehicle 350. In other embodiments, the autonomous vehicle control system may interface only with a teleassist system or only with a real-time map system, and the present invention is not limited to the specific embodiments described herein. Additionally, Figure 7 illustrates only those components of the autonomous vehicle control system 352 that are relevant to motion and path planning; other components unrelated to that functionality have been omitted from the drawing for simplicity.

[0096] One or more image sensors or cameras 354 and one or more LIDAR sensors 356 operate as sensors used in connection with motion planning. Camera 354 outputs camera data to a recognition component 358 and, in particular, to an object classification component 360, which receives the output of LIDAR sensor 356. Object classification component 360 can be used to identify physical objects present in the autonomous vehicle's environment. While component 360 can detect various types of physical objects, in terms of the functionality described herein, two specific types of objects are shown as the output of object classification component 360: construction elements and signs. However, the types of objects that can be detected are numerous, and in other embodiments, the recognition component can be configured to detect multiple other types of objects, so the construction elements and signs outputs shown in FIG. 7 are merely illustrative examples.

[0097] The recognition component 358 may include a sign classification component 362 that receives objects that are signs detected by the object classification component 360 and determines the logical meaning of these signs, such as the type of sign (speed limit, warning, road closure, construction notice, etc.), whether the sign is warning or should be obeyed, or the location and / or lanes affected by the sign (e.g., right lane closed ahead 2 miles), etc. In other embodiments, the functionality of components 360, 362 may be combined, or additional purpose-specific classification components may be used.

[0098] Objects detected by recognition component 358 are provided to tracking component 364, which maintains a trajectory of each detected object (and other fixed and / or moving objects detected in the environment) over time. Because autonomous vehicles are typically moving, it can be appreciated that trajectories can still be used to represent fixed objects, as the trajectory's position relative to the autonomous vehicle will change over time due to the movement of the autonomous vehicle itself.

[0099] The LIDAR sensor 356 may provide outputs to other components of the autonomous vehicle control system 352. For example, the output may also be provided to an online position estimation component 366, which determines the current position and orientation of the autonomous vehicle. A global pose component 368 receives the output of the LIDAR sensor 356 and determines a global pose of the autonomous vehicle, which is output to a pose filter component 370 and fed back as an additional input to the global pose component 368. In some embodiments, the global pose may not be used as an input to the lane alignment component 372, but the global pose may be output to a lane alignment component 372 that receives the output of the LIDAR sensor 356 and the output of the lane boundary component 374 as inputs and provides other inputs to the pose filter 370. The lane alignment may be an input used to determine the global pose in some embodiments.

[0100] Lane boundary component 374 receives the output of camera 354 and / or LIDAR sensor 356 and can be used to generate recognized lane boundaries based on the output of camera 354 and / or LIDAR sensor 356, for example. The lane boundaries can then be provided to lane alignment component 372 to refine the global pose based on the lane boundaries detected in the environment. In some embodiments, camera-based and LIDAR-based lane alignment models can be used separately, with the former having a wider range but less accuracy and the latter having a higher accuracy but shorter range. In some embodiments, the camera-based model can be used for purposes such as map error detection, and the LIDAR-based model can be used for lane alignment and global pose determination.

[0101] A map fusion component 376 receives as input the trajectory output by the tracking component 364 and the recognized lane boundaries output by the lane boundary component 374, where these trajectories and boundaries are considered to represent at least a portion of the recognized observation data collected by the autonomous vehicle's sensors. A road region generation component 378 also retrieves map data from an onboard map 380 and generates a base road region layout (RRL) representing a digital map representing the local road region around the autonomous vehicle. The map fusion component 376 fuses these inputs to generate an augmented road region layout (ARRL), which is provided to a motion planning component 382 that generates a motion path for the autonomous vehicle based at least in part on the augmented road region layout.

[0102] The motion planner component 382 may receive as input the portion of the trajectory output by the tracking component 364 and a desired route provided by the path planner component 384. This route is typically generated from the onboard map 308 and provides high-level guidance on the desired route toward the desired destination. In some embodiments, an additional occlusion area component 386 also receives the portion of the trajectory output by the tracking component 364 and the augmented road area layout provided by the map fusion component 376 to determine one or more occlusion areas in the environment, which are output as additional trajectories to the motion planner component 382. As previously mentioned, an occlusion area may represent a portion of the environment where drivability is not possible, and in some embodiments, may represent a collection of associated construction elements, such as cones, barrels, and / or blocking barriers, that may be considered to constitute a boundary that an autonomous vehicle cannot traverse.

[0103] The teleassist component 388 may also be operatively coupled to the map fusion component 376 and the motion planner component 382 to provide an interface with a remote teleassist system (not shown in FIG. 7 ). The teleassist component 388 may, for example, output observation data to the map fusion component 376 to be incorporated into the augmented road area. One non-limiting example is a speed limit, e.g., providing the maximum speed an autonomous vehicle may travel on a particular lane or road or within a construction zone. The teleassist component 388 may output suggested actions to the motion planner component 382, ​​such as changing lanes, stopping, or pulling over before a blocked area. The teleassist component 388 may receive the augmented road area layout (or other data collected by the map fusion component 376) to assist the remote teleassist driver in providing assistance to the autonomous vehicle during a teleassist session.

[0104] An online service 390, including a remote teleassist system and a real-time map system similar to those described above in connection with Figures 5-6, may interface with the map fusion component 376 and the teleassist component 388, for example, to provide observation data and / or location-based teleassist triggers to the map fusion component 376, or to exchange information with the teleassist driver during a teleassist session.

[0105] It will be appreciated that the architecture of autonomous vehicle control system 352 is merely exemplary in nature, and that other architectures may be used in other embodiments. For example, among other possible variations, occluded area component 386 may alternatively be implemented within management recognition component 358, such that occluded areas are managed as trajectories similar to construction elements and signs. It will also be understood that some or all of the components of autonomous vehicle control system 352 may be implemented using programmed logic and / or trained machine learning models, and that implementation of such components is well within the capabilities of one of ordinary skill in the art with reference to this disclosure.

[0106] FIG. 8 illustrates an exemplary embodiment of a real-time map system 400. The real-time map system 400 includes a real-time map database 402, a real-time map data collection component 404, a real-time map propagation component 406, a real-time map data quality assurance component 408, a real-time map training data generation component 410, and a teleassist interface component 412. The real-time map data collection component 404 can be used to receive observation data transmitted by one or more autonomous vehicles and collect the observation data into the real-time map database 402. Collection can include correlating new observation data with previously stored observation data and assigning or modifying expiration criteria for observation data to control when no longer observed observations are automatically removed from the database. The real-time map data propagation component 406 can be used to propagate observation data to autonomous vehicles in a fleet, for example, by an autonomous vehicle requesting observation data related to its current location or automatically prompted by the autonomous vehicle's tracked location. The map data propagation component 406 can propagate data to a map fusion component or directly to a teleassist component.

[0107] The real-time map data quality assurance component 408 can be used to validate observation data received from the autonomous vehicle. For example, it can be used to reject inaccurate or unreliable observation data, correct or supplement observation data, and propagate reliable observation data to the offline map system. Component 408 can provide an interface with a human driver to determine, for example, when observation data is reliable enough to be included in the offline map database. The real-time map training data generation component 410 can be used to generate training data used to train various machine learning models. The machine learning models include, for example, machine learning models implemented within the real-time map system 400 and / or within an autonomous vehicle control system. The teleassist interface 412 provides an interface with the teleassist system, for example, to enable the teleassist driver to generate position-based teleassist triggers or communicate observation data between the database 402 and the teleassist system.

[0108] Referring now to FIG. 9 , FIG. 9 illustrates an example operational sequence 420 that may be implemented within an autonomous vehicle control system to utilize a real-time map system and / or a teleassist system in connection with controlling an autonomous vehicle. As shown in block 422, a map fusion component may fuse various types of data to form an augmented road area layout. For example, recognition observations may be provided, for example, as detected by one or more sensors of the autonomous vehicle, in block 424, and real-time map data, such as observation data collected by other autonomous vehicles and / or location-based teleassist triggers, may be provided by the real-time map data propagation component 406, in block 426. Additionally, other teleassist data provided by the teleassist system may be provided, in block 428.

[0109] Next, motion planning may be performed based on the augmented road region layout to generate a motion plan (e.g., a motion path), for example, by a motion planner component, at block 430. The motion plan may then be used to control the autonomous vehicle, at block 432. Also, as shown at block 434, observational data collected from the autonomous vehicle, such as perception observational data collected from one or more sensors of the autonomous vehicle and / or the motion plan generated by the motion planner, may be transmitted to the real-time map data collection component 404 of the real-time map system.

[0110] Next, FIGS. 10-11 illustrate the generation and triggering of a location-based teleassist trigger, respectively. In particular, FIG. 10 illustrates an exemplary operational sequence 440 for generating and storing a location-based teleassist trigger in the real-time map system 400 of FIG. 8. In some embodiments, the operational sequence 440 may be performed at least in part by a teleassist system, such as the teleassist system 318 of FIG. 6, interfaced with the real-time map system 400 via the teleassist interface 412. In embodiments in which at least a portion of the operational sequence is performed in an autonomous vehicle, a map fusion component may perform at least some of the operations and, in some cases, may be used to trigger a teleassist session based on real-time map data received by the autonomous vehicle. As shown in blocks 442-446, a teleassist session is initiated (block 442), situational awareness data and teleassist driver input can be exchanged between the autonomous vehicle and the teleassist system during the session (block 444), and the teleassist session is then terminated at a particular point (block 446). Next, in block 448, the teleassist driver can selectively evaluate and / or modify the teleassist session data. For example, it may be determined whether any of the teleassisted session data should be included in the real-time map database, discarded, or otherwise processed.

[0111] Next, in block 450, it is determined whether to generate a location-based teleassist trigger for a teleassist session (either by the teleassist driver or, alternatively, in an automated manner by the teleassist system). For example, it may be determined that a particular road section, such as a construction zone, is generating teleassist session requests from multiple autonomous vehicles, and it may be preferable to trigger a new teleassist session for a vehicle approaching the section earlier so that the teleassist driver may have additional time to assist the autonomous vehicle. As another example, it may be determined that a particular maneuver proposed for a particular road section during a teleassist session or a particular route proposed during a teleassist session provided the best or at least adequate performance. This may then be communicated to other vehicles approaching the same section so that they perform the same maneuver or follow the same route. Potentially, this may be achieved without the need to initiate a separate teleassist session that would have been initiated without a proposal. As another example, certain types of signs and / or construction elements identified with a threshold confidence (by a single autonomous vehicle or multiple autonomous vehicles) may be used to identify particular road sections for which it may be preferable to pre-trigger a teleassist session for other autonomous vehicles.

[0112] If a location-based teleassist trigger is not proposed, sequence 440 may be completed. Otherwise, control proceeds to block 452 to generate a teleassist trigger, which may include one or more of location-based criteria, a session recommendation (e.g., to automatically trigger a session), a recommended action (e.g., to suggest an action for the autonomous vehicle to perform), and any related observations that may be useful to the autonomous vehicle and / or the driver of a later-initiated teleassist session. Related observations may include perception and / or operational observations, such as best or suboptimal actions taken, paths of other vehicles (autonomous and / or non-autonomous vehicles), etc.

[0113] FIG. 11 illustrates an exemplary operational sequence 460 for activating a location-based teleassist trigger using the autonomous vehicle control system and real-time map system illustrated in FIGS. 7-8. In particular, at block 462, the autonomous vehicle control system receives real-time map data for a current region from the real-time map system via the real-time map data propagation component 406. In this case, the real-time map data includes one or more location-based teleassist triggers associated with the autonomous vehicle's current region, instead of or in addition to the observational data. As illustrated at block 464, the autonomous vehicle then navigates using, in part, the real-time map data to determine whether the location-based criteria of the teleassist trigger are met based on the autonomous vehicle's position and / or path at the time represented at block 466. For example, a trigger may occur due to passing through a geofence, being within a predetermined distance of a particular location, being on a particular road or lane, etc.

[0114] At block 468, it is determined whether a teleassist session should be initiated, e.g., whether the location-based teleassist trigger has a positive session recommendation associated with it. If so, control passes to block 470 to automatically initiate the teleassist session. If not, control instead passes to block 472 to operate the autonomous vehicle according to one or more suggestions (e.g., a proposed action, a proposed path, etc.) associated with the teleassist trigger. It is also understood that in some embodiments, a trigger may include both a suggested action and a positive session recommendation, and blocks 470 and 472 may each be performed for a single teleassist trigger.

[0115] Next, FIG. 12 illustrates an example operational sequence 480 for controlling an autonomous vehicle based on other vehicle paths and / or suboptimal behavior using the autonomous vehicle control system and real-time map system of FIGS. 7-8. In particular, at block 482, the autonomous vehicle control system receives real-time map data for a current region from the real-time map system via real-time map data propagation component 406. At block 484, one or more vehicle paths stored as observations in the real-time map data may be added to an enhanced road region layout used by the motion planner. These paths may include, for example, paths used by other autonomous vehicles when traversing a particular road segment, which may provide the motion planner with clues about how the other autonomous vehicles traveled the same segment. These paths may include paths of other vehicles (including non-autonomous vehicles) detected by sensors on the other autonomous vehicles instead of, or in addition to, paths generated by the other autonomous vehicles. Thus, for example, if a particular autonomous vehicle is following one or more human-driven vehicles through a section of road, a trajectory showing the path of the vehicle tracked via the autonomous vehicle's perception system may provide clues to the motion planner as to how the human drivers would choose to traverse the same section, allowing the motion planner to select a motion plan that preferably mimics the path of the one or more other vehicles based on real-time map data.

[0116] Additionally, one or more suboptimal actions taken by other autonomous vehicles may be accessed from the real-time map data at block 486. Thus, for example, if another autonomous vehicle had to suddenly stop, suddenly slow down, pull over, or suddenly change direction while traversing the same road segment, the motion planner may choose to reject routes that may have produced similar results for the autonomous vehicle. Thus, at block 488, the motion planner may utilize the added routes and / or suboptimal actions to generate a motion plan suitable for the autonomous vehicle.

[0117] Next, Figures 13 and 14 illustrate an example of using the autonomous vehicle control system and real-time map system of Figures 7 and 8 to fuse real-time map data with a road area layout to generate an augmented road area layout. For example, Figure 13 shows a road area layout 500 generated from offline map data for an exemplary four-lane divided highway. This layout includes northbound lane pair 502, 504, southbound lane pair 506, 508, shoulder 510, and median 512. Two dividers 514, 516 are separated by gap 518 to allow police or emergency vehicles to pass, but this gap is not considered a drivable area for general vehicles according to law or regulation. Each arrowhead for lanes 502-508 also indicates logical map data defining the normal direction of travel for each lane.

[0118] FIG. 14 illustrates an augmented road region layout 520 generated by fusing observational data obtained from a real-time map system with the road region layout 500 of FIG. 13 . For example, the augmented road region layout may be the result of a construction zone being established that (a) closes lanes 502 and 504, (b) directs northbound traffic to lane 508 via gap 518, and (c) requires all southbound traffic to remain in lane 506. For example, observational data, such as a blocked area 522 associated with multiple construction elements 524, may be fused into the augmented road region layout 520. The blocked area 522 and / or construction elements 524 may, in some cases, be recognized observations collected from the autonomous vehicle as the controlled autonomous vehicle passes through the construction zone, or alternatively, may be observational data collected by other autonomous vehicles and propagated through the real-time map system. However, it will be appreciated that other observational data, such as construction element 526, may not be recognized by the controlled autonomous vehicle as it passes through the construction zone and may represent observational data collected from other autonomous vehicles.

[0119] The fused observational data can also be used to overwrite the offline map data, for example, opening interval 518 to traffic, closing lanes 502 and 504 to traffic, and reversing the direction of travel of lane 508.

[0120] The observational data may also include paths 528 of other autonomous and / or non-autonomous vehicles that have passed through the construction zone, which the motion planner can use to select movement paths similar to the paths of other vehicles. The observational data may also include suboptimal maneuvers that require abrupt changes in direction and speed reductions (shown at 532) when passing through the construction zone, as represented by path 530. The lane movements can be used by the motion planner to select movement paths that do not replicate the adverse consequences of the lane movements.

[0121] Although particular features may be described herein with reference to particular embodiments and / or particular drawings, it will be understood that such features may generally be included in all embodiments described and illustrated herein, unless expressly stated otherwise. Furthermore, features disclosed in combination in some embodiments may generally be implemented separately in other embodiments, and features implemented separately in some embodiments may be combined in other embodiments. Therefore, the fact that particular features are described in one embodiment but not in another should not be construed as an admission that the two embodiments are mutually exclusive. Other variations will be apparent to those of ordinary skill in the art. Accordingly, the scope of the present invention resides in the appended claims.

Claims

1. An autonomous vehicle control system, comprising: one or more processors; and a memory storing instructions that, when executed by the one or more processors, cause the autonomous vehicle control system to: receiving a digital map of a portion of an environment in which the autonomous vehicle is to operate, the digital map defining a plurality of static elements within the portion of the environment; receiving observation data relating to one or more observations collected from the environment from a remote real-time map system; augmenting the digital map with the observation data to generate an augmented digital map; An autonomous vehicle control system that controls the autonomous vehicle using the augmented digital map.

2. The one or more processors: determining a trajectory of the autonomous vehicle using the augmented digital map; The autonomous vehicle control system of claim 1 , configured to control the autonomous vehicle using the augmented digital map by controlling the autonomous vehicle along the trajectory.

3. The observation data is first observation data, and the one or more processors: receiving second observational data collected using one or more sensors of the autonomous vehicle; 2. The autonomous vehicle control system of claim 1, further configured to augment the digital map with the second observation data, the augmented digital map being augmented using the first observation data and the second observation data.

4. 4. The autonomous vehicle control system of claim 3, wherein the one or more processors are further configured to forward the second observation data to the remote real-time map system for use in controlling one or more other autonomous vehicles operating in the environment.

5. 2. The autonomous vehicle control system of claim 1, wherein the one or more observations collected from the environment are collected by one or more other autonomous vehicles operating within the environment and forwarded to the remote real-time map system.

6. The autonomous vehicle control system of claim 1 , wherein the observational data defines construction elements detected in the environment. In some embodiments, the observational data defines occlusion areas within the environment that restrict movement of vehicles within the environment.

7. The autonomous vehicle control system of claim 1 , wherein the observational data defines occlusion areas within the environment that restrict vehicle movement within the environment.

8. 2. The autonomous vehicle control system of claim 1, wherein the observational data defines one or more vehicle paths, each vehicle path being related to detected paths of other vehicles while traveling through the environment, and the one or more processors are configured to control the autonomous vehicle using the augmented digital map by determining a movement path for the autonomous vehicle using the one or more vehicle paths.

9. 2. The autonomous vehicle control system of claim 1, wherein the observational data defines a suboptimal action to be taken by another autonomous vehicle while navigating the environment, and the one or more processors are configured to control the autonomous vehicle using the augmented digital map by determining a movement path for the autonomous vehicle using the suboptimal action.

10. The one or more processors: receiving a location-based teleassist trigger from the remote real-time map system; The autonomous vehicle control system of claim 1 , further configured to determine activation of the position-based teleassist trigger based at least in part on position-based criteria associated with the position-based teleassist trigger.

11. 11. The autonomous vehicle control system of claim 10, wherein the one or more processors are further configured to automatically initiate a teleassist session with a remote teleassist system in response to determining activation of the position-based teleassist trigger.

12. 11. The autonomous vehicle control system of claim 10, wherein the one or more processors are further configured to, in response to determining activation of the position-based teleassist trigger, control the autonomous vehicle based at least in part on a proposed action associated with the position-based teleassist trigger.

13. 13. The autonomous vehicle control system of claim 12, wherein the one or more processors are configured to control the autonomous vehicle based at least in part on the proposed action associated with the location-based teleassist trigger without initiating a teleassist session with a remote teleassist system.

14. The autonomous vehicle control system of claim 13 , wherein the proposed action is received from the remote real-time map system and generated from another teleassist session with another autonomous vehicle.

15. A method for operating an autonomous vehicle using an autonomous vehicle control system includes: receiving a digital map of a portion of an environment in which the autonomous vehicle is to operate, the digital map defining a plurality of static elements within the portion of the environment; receiving observation data relating to one or more observations collected from the environment from a remote real-time map system; augmenting the digital map with the observation data to generate an augmented digital map; and controlling the autonomous vehicle using the augmented digital map.

16. An autonomous vehicle control system, comprising: one or more processors; and a memory storing instructions that, when executed by the one or more processors, cause the autonomous vehicle control system to: receiving a digital map of a portion of an environment in which the autonomous vehicle is to operate, the digital map defining a plurality of static elements within the portion of the environment; determining observation data related to one or more observations collected from the environment based on sensor data collected using one or more sensors of the autonomous vehicle; augmenting the digital map with the observation data to generate an augmented digital map; controlling the autonomous vehicle using the augmented digital map; An autonomous vehicle control system that causes the observational data to be transferred to a remote real-time mapping system for use in controlling one or more other autonomous vehicles operating within the environment.

17. The one or more processors: determining a trajectory of the autonomous vehicle using the augmented digital map; 17. The autonomous vehicle control system of claim 16, configured to control the autonomous vehicle using the augmented digital map by controlling the autonomous vehicle along the trajectory.

18. The observation data is first observation data, and the one or more processors: receiving second observation data from the remote real-time map system relating to one or more observations collected from the environment; 17. The autonomous vehicle control system of claim 16, further configured to augment the digital map with the second observation data, such that the augmented digital map is augmented using the first and second observation data.

19. The autonomous vehicle control system of claim 16 , wherein the observational data defines construction elements detected in the environment using sensors of the autonomous vehicle.

20. The autonomous vehicle control system of claim 16 , wherein the observational data defines occlusion areas within the environment that restrict vehicle movement within the environment.

21. 17. The autonomous vehicle control system of claim 16, wherein the observational data defines one or more vehicle paths, each vehicle path being related to a detected path of another vehicle or the path of the autonomous vehicle while navigating the environment, and wherein the one or more processors are configured to forward the observational data to the remote real-time map system by forwarding the one or more vehicle paths to the remote real-time map system for use in controlling the one or more other autonomous vehicles operating within the environment.

22. 17. The autonomous vehicle control system of claim 16, wherein the observational data defines a sub-optimal action to be taken by the autonomous vehicle while navigating the environment, and the one or more processors are configured to forward the observational data to the remote real-time mapping system for use in controlling the one or more other autonomous vehicles operating within the environment by forwarding the sub-optimal action to the remote real-time mapping system.

23. 1. A method of operating an autonomous vehicle using an autonomous vehicle control system, comprising: receiving a digital map of a portion of an environment in which the autonomous vehicle is to operate, the digital map defining a plurality of static elements within the portion of the environment; determining observation data related to one or more observations collected from the environment based on sensor data collected using one or more sensors of the autonomous vehicle; augmenting the digital map with the observation data to generate an augmented digital map; controlling the autonomous vehicle using the augmented digital map; and transmitting the observational data to a remote real-time mapping system for use in controlling one or more other autonomous vehicles operating in the environment.

24. 1. A method of operating an autonomous vehicle, comprising: receiving observation data relating to one or more observations collected from the environment from a remote real-time map system, the observation data defining a plurality of vehicle paths, each vehicle path relating to a detected path of another vehicle while traveling through the environment; planning a movement using the received observational data to generate a movement path for the autonomous vehicle based at least in part on the plurality of vehicle paths; and controlling the autonomous vehicle using the motion path.

25. 25. The method of claim 24, wherein a first vehicle path of the plurality of vehicle paths is associated with a detected path of a first vehicle detected using one or more sensors of a second vehicle operating within the environment.

26. 26. The method of claim 25, wherein the first vehicle is a non-autonomous vehicle.

27. The method of claim 24 , wherein a plurality of the vehicle paths of the plurality of vehicle paths are associated with non-autonomous vehicles.

28. 25. The method of claim 24, wherein the observational data further includes sub-optimal actions performed by other autonomous vehicles while navigating the environment, and wherein planning a movement using the received observational data to generate the movement path for the autonomous vehicle is further based at least in part on the sub-optimal actions.

29. 1. A method comprising: conducting a teleassist session with a first autonomous vehicle operating within an environment, the teleassist session including exchanging situational data and teleassist driver inputs between the first autonomous vehicle and a remote teleassist system; generating a location-based teleassist trigger in association with the teleassist session; forwarding the location-based teleassist trigger to a remote real-time map system, and causing the remote real-time map system to forward the location-based teleassist trigger to a second autonomous vehicle.

30. 30. The method of claim 29, wherein the location-based teleassist trigger includes a session proposal that selectively proposes automatic initiation of a teleassist session with the second autonomous vehicle when the second autonomous vehicle meets location-based criteria associated with the location-based teleassist trigger.

31. 30. The method of claim 29, wherein the location-based teleassist trigger includes a proposed action to be performed by the second autonomous vehicle when the second autonomous vehicle satisfies a location-based criterion associated with the location-based teleassist trigger.

32. 30. The method of claim 29, further comprising: receiving, at the first autonomous vehicle, observational data relating to one or more observations collected from the environment from the remote real-time map system; and controlling the autonomous vehicle using the received observational data.

33. 30. The method of claim 29, further comprising selectively propagating collected observation data associated with the remote teleassist system to the remote real-time map system in response to teleassist driver input.

34. 1. A method comprising: receiving observation data from a remote real-time map system relating to one or more observations collected from an environment in which the first autonomous vehicle is operating; controlling the first autonomous vehicle using the observation data; receiving, from the remote real-time map system, a location-based teleassist trigger generated in association with a teleassist session conducted between the remote teleassist system and a second autonomous vehicle; determining activation of the position-based teleassist trigger in response to determining that the first autonomous vehicle satisfies position-based criteria associated with the position-based teleassist trigger.

35. the teleassisted session is a first teleassisted session, and the location-based teleassisted trigger includes a session proposal that selectively proposes automatic initiation of a second teleassisted session; 35. The method of claim 34, further comprising automatically initiating the second teleassist session between the remote teleassist system and the first autonomous vehicle when the first autonomous vehicle satisfies the location-based criteria associated with the location-based teleassist trigger.

36. 35. The method of claim 34, wherein the position-based teleassist trigger includes a proposed action to be performed by the first autonomous vehicle, the method further including controlling the first autonomous vehicle based at least in part on the proposed action.

37. 1. A real-time map system, comprising: one or more processors; and a memory storing instructions that, when executed by the one or more processors, cause the real-time map system to: maintaining observation data relating to a plurality of observations collected by a plurality of autonomous vehicles operating within the environment; maintaining a location-based teleassist trigger generated in association with a teleassist session between a remote teleassist system and a first autonomous vehicle of the plurality of autonomous vehicles; and a real-time map system that transmits a portion of the observation data and the location-based teleassist trigger to a second autonomous vehicle of the plurality of autonomous vehicles for use in controlling the second autonomous vehicle.

Images