Remote real-time map system for autonomous vehicles

Abstract

A real-time map system may be used to propagate observations collected by an autonomous vehicle operating in an environment to other autonomous vehicles to supplement a digital map used in the control of the autonomous vehicle. Further, in some examples, a real-time map system may be used to propagate location-based remote assistance triggers to autonomous vehicles operating within an environment. A location-based remote assistance trigger may be generated, for example, in association with a remote assistance session made between an autonomous vehicle and a remote remote assistance system in the vicinity of a particular location, and may be used to automatically trigger a remote assistance session of another autonomous vehicle in the vicinity of the location and / or to propagate a suggested action to the other autonomous vehicle.

Description

Background Art

[0001] As computing and vehicle technologies continue to evolve, autonomy-related features have become more powerful and widely available and are capable of controlling vehicles in a wider range of situations. For example, for automobiles, the automotive industry generally adopts the SAE International standard J3016, which specifies six levels of autonomy. Vehicles without autonomy are designated as level 0, and in the case of level 1 autonomy, the vehicle controls steering or speed (but not both), leaving the operator to perform most vehicle functions. In the case of level 2 autonomy, the vehicle is able to control steering, speed, and braking in limited situations (e.g., when driving along a highway), but the operator is still required to remain alert and be ready to take over the operation at any time and handle any maneuvers, such as changing lanes or turning. Starting from level 3 autonomy, the vehicle is able to manage most operating variables, including monitoring the surrounding environment, but the operator is still required to remain alert and take over whenever a scenario that the vehicle cannot handle is encountered. Level 4 autonomy provides the ability to operate without operator input only under specific conditions, such as only on certain types of roads (e.g., highways) or only in certain geographical areas (e.g., specific cities with sufficient map data). Finally, level 5 autonomy represents the level of autonomy at which a vehicle is able to operate without operator control in any situation in which a human operator could also operate.

[0002] A fundamental challenge for any autonomy-related technology involves collecting and interpreting information about the vehicle's surrounding environment and making and implementing decisions to appropriately control the vehicle given the current environment in which the vehicle is operating. Accordingly, continuous efforts are being made to improve each of these aspects, and by doing so, autonomous vehicles are increasingly able to reliably handle a greater variety of situations and adapt to expected and unexpected conditions within the environment.

[0003] For example, a particular challenge is due to the inherently dynamic environment in which autonomous vehicles are expected to operate. For example, many autonomous vehicles rely on high-resolution digital maps representing various static objects in the environment, static objects including, for example, real-world objects or elements such as roads, curbs, buildings, trees, signs, etc., and logical elements such as lanes, boundaries, etc. Generally, attempts are made to maintain and update high-quality digital maps to accommodate changes that occur in the environment; however, the overhead associated with validating map data and distributing it to a fleet of autonomous vehicles can be significant. Additionally, even with rapid updates, changes can still suddenly occur in the environment and not be reflected in the map data used by autonomous vehicles operating in the environment.

[0004] In addition, although in some instances, the perception system of an autonomous vehicle can be used to detect changing situations in the environment (e.g., the presence of new construction elements such as cones and / or barrels), the detection range of such a system is typically limited and may be occluded by the presence of other vehicles nearby, such that the amount of time that can be given for the autonomous vehicle to react to some changing situations may be undesirably short. Accordingly, there is a continuing need in the art for ways to improve an autonomous vehicle's awareness of relevant objects and elements in its environment. SUMMARY OF THE INVENTION

[0005] The present disclosure relates in part to using a real-time map system to propagate observations collected by an autonomous vehicle operating in an environment to other autonomous vehicles, and thereby supplement a digital map used in the control of the autonomous vehicle. Additionally, in some instances, the real-time map system can be used to propagate location-based remote assistance triggers to autonomous vehicles operating within the environment. A location-based remote assistance trigger can be generated, for example, in association with a remote assistance session conducted between an autonomous vehicle and a distal remote assistance system near a particular location, and can be used to automatically trigger a remote assistance session of another autonomous vehicle near that location and / or propagate a suggested action to that other autonomous vehicle.

[0006] Accordingly, in one aspect of the invention, an autonomous vehicle control system can include: one or more processors; and a memory storing instructions that, when executed by the one or more processors, cause the autonomous vehicle control system to receive a digital map of a portion of the environment in which the autonomous vehicle operates, the digital map defining a plurality of static elements within the portion of the environment, receive observation data associated with one or more observations collected from the environment from a distal real-time map system, enhance the digital map with the observation data to generate an enhanced digital map, and use the enhanced digital map to control the autonomous vehicle.

[0007] In some embodiments, the one or more processors are configured to use the enhanced digital map to control the autonomous vehicle by determining a trajectory of the autonomous vehicle using the enhanced digital map and controlling the autonomous vehicle according to the trajectory.

[0008] In addition, in some embodiments, the observation data is first observation data, and the one or more processors are further configured to receive second observation data collected using one or more sensors of the autonomous vehicle, and enhance the digital map with the second observation data such that the enhanced digital map is enhanced using the first observation data and the second observation data. Additionally, in some embodiments, the one or more processors are further configured to transmit the second observation data to the distal real-time map system for use in controlling one or more other autonomous vehicles operating in the environment.

[0009] In some embodiments, one or more observations collected from the environment are collected by one or more other autonomous vehicles operating in the environment and transmitted to a remote real-time map system. Additionally, in some embodiments, the observation data defines construction elements detected in the environment. In some embodiments, the observation data defines blocked areas in the environment that restrict vehicle movement within the environment. Additionally, in some embodiments, the observation data defines one or more vehicle paths, each vehicle path associated with the detected path of a different vehicle as it travels through the environment, and one or more processors are configured to control an autonomous vehicle by using the enhanced digital map to determine a movement path of the autonomous vehicle by using the one or more vehicle paths.

[0010] Additionally, in some embodiments, the observation data defines suboptimal actions performed by another autonomous vehicle as it travels through the environment, and one or more processors are configured to control an autonomous vehicle by using the enhanced digital map to determine a movement path of the autonomous vehicle by using the suboptimal actions. In some embodiments, one or more processors are further configured to receive a location-based remote assistance trigger from the remote real-time map system and determine the activation of the location-based remote assistance trigger at least in part based on location-based criteria associated with the location-based remote assistance trigger.

[0011] Additionally, in some embodiments, one or more processors are further configured to automatically initiate a remote assistance session with a remote remote assistance system in response to determining the activation of the location-based remote assistance trigger. In some embodiments, one or more processors are further configured to control an autonomous vehicle at least in part based on recommended actions associated with the location-based remote assistance trigger in response to determining the activation of the location-based remote assistance trigger. Additionally, in some embodiments, one or more processors are configured to control an autonomous vehicle at least in part based on recommended actions associated with the location-based remote assistance trigger without initiating a remote assistance session with the remote remote assistance system. In some embodiments, the recommended actions are received from the remote real-time map system and are generated based on different remote assistance sessions with different autonomous vehicles.

[0012] According to another aspect of the present invention, a method of operating an autonomous vehicle with an autonomous vehicle control system may include: receiving a digital map of a portion of an environment in which the autonomous vehicle operates, the digital map defining a plurality of static elements within the portion of the environment; receiving observation data associated with one or more observations collected from the environment from a remote real-time map system; enhancing the digital map with the observation data to generate an enhanced digital map; and using the enhanced digital map to control the autonomous vehicle.

[0013] According to another aspect of the present invention, an autonomous vehicle control system may include one or more processors; and a memory storing instructions that, when executed by the one or more processors, cause the autonomous vehicle control system to receive a digital map of a portion of the environment in which the autonomous vehicle operates, the digital map defining a plurality of static elements within the portion of the environment, determine observation data associated with one or more observations collected from the environment based on sensor data collected by one or more sensors of the autonomous vehicle, enhance the digital map with the observation data to generate an enhanced digital map, use the enhanced digital map to control the autonomous vehicle, and transmit the observation data to a remote real-time map system for controlling one or more other autonomous vehicles operating in the environment.

[0014] In addition, in some embodiments, the one or more processors are configured to use the enhanced digital map to control the autonomous vehicle by determining a trajectory of the autonomous vehicle using the enhanced digital map and controlling the autonomous vehicle according to the trajectory.

[0015] In addition, in some embodiments, the observation data is first observation data, and the one or more processors are further configured to receive second observation data associated with one or more observations collected from the environment from the remote real-time map system, and enhance the digital map with the second observation data such that the enhanced digital map is enhanced with the first observation data and the second observation data. In some embodiments, the observation data defines construction elements detected in the environment using sensors of the autonomous vehicle. In addition, in some embodiments, the observation data defines a blocked area in the environment that restricts vehicle movement within the environment.

[0016] In addition, in some embodiments, the observation data defines one or more vehicle paths, each vehicle path associated with a path detected when a different vehicle travels through the environment or the path of the autonomous vehicle, and the one or more processors are configured to transmit the observation data to the remote real-time map system by transmitting the one or more vehicle paths to the remote real-time map system for controlling one or more other autonomous vehicles operating in the environment.

[0017] In addition, in some embodiments, the observation data defines sub-optimal actions performed by the autonomous vehicle when traveling through the environment, and the one or more processors are configured to transmit the observation data to the remote real-time map system by transmitting the sub-optimal actions to the remote real-time map system for controlling one or more other autonomous vehicles operating in the environment.

[0018] According to another aspect of the present invention, a method of operating an autonomous vehicle may include: receiving a digital map of a portion of the environment in which the autonomous vehicle operates, the digital map defining a plurality of static elements within the portion of the environment; determining observation data associated with one or more observations collected from the environment based on sensor data collected by one or more sensors of the autonomous vehicle; enhancing the digital map with the observation data to generate an enhanced digital map; using the enhanced digital map to control the autonomous vehicle; and transmitting the observation data to a remote real-time map system for controlling one or more other autonomous vehicles operating in the environment.

[0019] According to another aspect of the present invention, a method of operating an autonomous vehicle may include: receiving, from a remote real-time map system, observation data associated with one or more observations collected from the environment, wherein the observation data defines a plurality of vehicle paths, each vehicle path associated with a detected path of a different vehicle as it travels through the environment; using the received observation data for motion planning to generate a motion path of the autonomous vehicle based at least in part on the plurality of vehicle paths; and using the motion path to control the autonomous vehicle.

[0020] In addition, in some embodiments, a first vehicle path among the plurality of vehicle paths is associated with a detected path of a first vehicle detected using one or more sensors of a second vehicle operating in the environment. In addition, in some embodiments, the first vehicle is a non-autonomous vehicle. In addition, in some embodiments, a plurality of the vehicle paths among the plurality of vehicle paths are associated with non-autonomous vehicles. In some embodiments, the observation data further includes sub-optimal actions performed by another autonomous vehicle as it travels through the environment, and using the received observation data for motion planning to generate a motion path of the autonomous vehicle is further based at least in part on the sub-optimal actions.

[0021] According to another aspect of the present invention, a method may include conducting a remote assistance session with a first autonomous vehicle operating in an environment, including exchanging situational data and remote assistance operator input between the first autonomous vehicle and a remote remote assistance system, generating a location-based remote assistance trigger associated with the remote assistance session, and transmitting the location-based remote assistance trigger to a remote real-time map system to cause the remote real-time map system to forward the location-based remote assistance trigger to a second autonomous vehicle.

[0022] In addition, in some embodiments, location-based remote assistance triggering includes a session suggestion that selectively suggests an automatic initiation of a remote assistance session with a second autonomous vehicle when the second autonomous vehicle meets a location-based criterion associated with the location-based remote assistance triggering. In some embodiments, location-based remote assistance triggering includes a suggested action to be taken by the second autonomous vehicle when the second autonomous vehicle meets a location-based criterion associated with the location-based remote assistance triggering.

[0023] Some embodiments may further include: in a first autonomous vehicle, receiving observation data associated with one or more observations collected from the environment from a remote real-time map system and using the received observation data to control the autonomous vehicle. Some embodiments may further include selectively propagating the observation data collected in association with a remote assistance system at a remote end to the remote real-time map system in response to a remote assistance operator input.

[0024] According to another aspect of the present invention, a method may include: receiving observation data associated with one or more observations collected from the environment in which a first autonomous vehicle operates from a remote real-time map system; controlling the first autonomous vehicle using the observation data; receiving a location-based remote assistance trigger generated in association with a remote assistance session between a remote assistance system at a remote end and a second autonomous vehicle from the remote real-time map system; and determining an activation of the location-based remote assistance trigger in response to determining that the first autonomous vehicle meets a location-based criterion associated with the location-based remote assistance trigger.

[0025] In some embodiments, the remote assistance session is a first remote assistance session, the location-based remote assistance trigger includes a session suggestion that selectively suggests an automatic initiation of a second remote assistance session, and the method further includes automatically initiating a second remote assistance session between the remote assistance system at the remote end and the first autonomous vehicle when the first autonomous vehicle meets a location-based criterion associated with the location-based remote assistance trigger. In addition, in some embodiments, the location-based remote assistance trigger includes a suggested action to be taken by the first autonomous vehicle, and the method further includes controlling the first autonomous vehicle at least in part based on the suggested action.

[0026] According to another aspect of the present invention, a real-time map system may include one or more processors; and a memory storing instructions that, when executed by the one or more processors, cause the real-time map system to maintain observation data associated with a plurality of observations collected by a plurality of autonomous vehicles operating in an environment, maintain location-based remote assistance triggers generated in association with a remote assistance session conducted between a remote assistance system at a remote location and a first autonomous vehicle among the plurality of autonomous vehicles, and transmit a portion of the observation data and the location-based remote assistance triggers to a second autonomous vehicle among the plurality of autonomous vehicles for controlling the second autonomous vehicle.

[0027] Some embodiments may also include autonomous vehicles and / or systems located away from the autonomous vehicles, and include one or more processors configured to perform the various methods described above. Some embodiments may also include an autonomous vehicle control system that includes one or more processors, a computer-readable storage medium, and computer instructions residing on the computer-readable storage medium and executable by the one or more processors to perform the various methods described above. Still other embodiments may include a non-transitory computer-readable storage medium that stores computer instructions executable by one or more processors to perform the various methods described above.

[0028] It should be understood that all combinations of the foregoing concepts and additional concepts described in greater detail herein are considered to be part of the subject matter disclosed herein. For example, all combinations of the claimed subject matter appearing at the end of this disclosure are contemplated to be part of the subject matter disclosed herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 Illustrates an example hardware and software environment for an autonomous vehicle.

[0030] Figure 2 Is a block diagram illustrating an example remote assistance enabled system consistent with some embodiments.

[0031] Figure 3 Is a further illustration Figure 2 Of the block diagram of the remote assistance control module referenced in

[0032] Figure 4 Is a block diagram illustrating the interaction of an autonomous vehicle with an example real-time map and remote assistance system consistent with some embodiments.

[0033] Figure 5 Is a block diagram illustrating a system for enabling real-time map and remote assistance for controlling an autonomous vehicle consistent with some embodiments.

[0034] Figure 6 Is a block diagram illustrating another system for enabling real-time map and remote assistance for controlling an autonomous vehicle consistent with some embodiments.

[0035] Figure 7 is a block diagram illustrating an example real-time map and a remote assistance-enabled autonomous vehicle control system consistent with some embodiments.

[0036] Figure 8 is a block diagram illustrating a real-time map system consistent with some embodiments and capable of interacting with Figure 7 an autonomous vehicle control system.

[0037] Figure 9 is a flowchart illustrating an example sequence of operations for controlling an autonomous vehicle using Figures 7 to 8 an autonomous vehicle control system and a real-time map system.

[0038] Figure 10 is a flowchart illustrating an example sequence of operations for generating and storing location-based remote assistance triggers in Figure 8 a real-time map system.

[0039] Figure 11 is a flowchart illustrating an example sequence of operations for activating location-based remote assistance triggers using Figures 7 to 8 an autonomous vehicle control system and a real-time map system.

[0040] Figure 12 is a flowchart illustrating an example sequence of operations for controlling an autonomous vehicle based on other vehicle paths and / or sub-optimal actions using Figures 7 to 8 an autonomous vehicle control system and a real-time map system.

[0041] Figure 13 and Figure 14 are block diagrams illustrating an example fusion of real-time map data with a road area layout to generate an enhanced road area layout using Figures 7 to 8 an autonomous vehicle control system and a real-time map system. Detailed Description

[0042] The various embodiments discussed below generally relate in part to a real-time map system for use with an autonomous vehicle to supplement a digital map. However, before discussing these embodiments, an example hardware and software environment in which the various techniques disclosed herein may be implemented will be discussed.

[0043] Hardware and software environment

[0044] Turning to the drawings, in which like numerals represent like parts throughout the several views, Figure 1FIG. illustrates an example autonomous vehicle 100 in which the various techniques disclosed herein may be implemented. For example, vehicle 100 is shown traveling on road 101, and vehicle 100 may include: a powertrain 102 that includes a prime mover 104 powered by an energy source 106 and capable of providing power to a driveline 108; and a control system 110 that includes a steering control 112, a powertrain control 114, and a braking control 116. Vehicle 100 may be implemented as any number of different types of vehicles, including vehicles capable of transporting people and / or cargo and capable of traveling on land, sea, air, underground, underwater, and / or in space, and it should be understood that the foregoing components 102-116 may vary widely based on the type of vehicle in which they are utilized.

[0045] For example, the embodiments discussed below will focus on wheeled land vehicles such as cars, vans, trucks, buses, etc. In such embodiments, prime mover 104 may include one or more electric motors and / or internal combustion engines (among others), while energy source 106 may include a fuel system (e.g., providing gasoline, diesel, hydrogen, etc.), a battery system, solar panels or other renewable energy sources, a fuel cell system, etc., and driveline 108 may include wheels and / or tires and a transmission and / or any other mechanical drive components suitable for converting the output of prime mover 104 into vehicle motion, as well as one or more brakes configured to controllably stop or slow the vehicle, and a direction or steering component suitable for controlling the trajectory of the vehicle (e.g., a rack and pinion steering linkage enables one or more wheels of vehicle 100 to pivot about a generally vertical axis to change the angle of the rotational plane of the wheels relative to the longitudinal axis of the vehicle). In some embodiments, for example, in the case of an electric / gas hybrid vehicle, a combination of powertrain and energy source may be used, and in some instances, multiple electric motors (e.g., dedicated to individual wheels or axles) may be used as the prime mover. In the case of a hydrogen fuel cell embodiment, the prime mover may include one or more electric motors, and the energy source may include a fuel cell system powered by hydrogen fuel.

[0046] Steering control 112 may include one or more actuators and / or sensors for controlling and receiving feedback from the direction or steering component to enable the vehicle to follow a desired trajectory. Powertrain control 114 may be configured to control the output of powertrain 102, e.g., control the output power of prime mover 104, control the gears of the transmission in driveline 108, etc., thereby controlling the speed and / or direction of the vehicle. Braking control 116 may be configured to control one or more brakes that decelerate or stop vehicle 100, e.g., disc or drum brakes coupled to the wheels of the vehicle.

[0047] Other vehicle types, including but not limited to off-road vehicles, all-terrain or tracked vehicles, construction equipment, etc., will necessarily utilize different powertrains, drivetrains, energy sources, steering controls, powertrain controls, and braking controls, as will be understood by one of ordinary skill in the art who benefits from this disclosure. Additionally, in some embodiments, some components may be combined, for example, where the steering control of the vehicle is handled primarily by varying the output of one or more prime movers. Accordingly, the present invention is not limited to the specific application of the techniques described herein to autonomous wheeled land vehicles.

[0048] In the illustrated embodiment, the autonomous control of vehicle 100 (which may include various levels of autonomy as well as selective autonomous functions) is implemented primarily in the main vehicle control system 120, which may include one or more processors 122 and one or more memories 124, where each processor 122 is configured to execute program code instructions 126 stored in the memory 124.

[0049] The main sensor system 130 may include a variety of sensors suitable for collecting information from the vehicle's surrounding environment for controlling the operation of the vehicle. For example, a satellite navigation (SATNAV) sensor 132, which is compatible with any one of various satellite navigation systems such as GPS, GLONASS, Galileo, Compass, etc., may be used to determine the vehicle's position on the earth using satellite signals. Radio detection and ranging (RADAR) and light detection and ranging (LIDAR) sensors 134, 136, and a digital camera 138 (which may include various types of image capture devices capable of capturing still and / or video images) may be used to sense stationary and moving objects in the vicinity of the vehicle. An inertial measurement unit (IMU) 140 may include multiple gyroscopes and accelerometers capable of detecting the linear and rotational motion of the vehicle in three directions, while one or more wheel encoders 142 may be used to monitor the rotation of one or more wheels of vehicle 100.

[0050] The outputs of sensors 132-142 can be provided to a set of master control subsystems 150, including a localization subsystem 152, a planning subsystem 154, a perception subsystem 156, and a control subsystem 158. The localization subsystem 152 is primarily responsible for precisely determining the position and orientation of vehicle 100 within its surrounding environment and typically within a certain reference frame (sometimes also referred to as “pose,” which may also include one or more velocities and / or accelerations in some instances). The planning subsystem 154 is primarily responsible for planning the motion path of vehicle 100 within a certain time frame given a desired destination and static and moving objects within the environment, while the perception subsystem 156 is primarily responsible for detecting, tracking, and / or identifying elements within the surrounding environment of vehicle 100. The control subsystem 158 is primarily responsible for generating appropriate control signals for controlling the various controls in control system 110 to achieve the planned path of the vehicle. Any one or all of the localization subsystem 152, the planning subsystem 154, the perception subsystem 156, and the control subsystem 158 may have associated data that is generated and / or utilized in conjunction with their operation and may be transmitted to a remote assistance system in some embodiments.

[0051] In addition, in the illustrated embodiments, a map or atlas subsystem 160 may be provided to describe elements within the environment and their relationships. The atlas subsystem 160 may be accessed by each of the localization, planning, and perception subsystems 152 - 156 to obtain various information about the environment for performing their respective functions. The atlas subsystem 160 may be used to provide map data to the autonomous vehicle control system, which may be used for various purposes in the autonomous vehicle, including for localization, planning, and perception purposes. The map data may be used, for example, to layout or place elements within a particular geographic region, including elements representing real-world objects such as roads, boundaries (e.g., obstacles, lane dividers, medians, etc.), buildings, traffic devices (e.g., traffic or road signs, lights, etc.), and elements that are more logical or virtual in nature, such as elements representing valid paths that a vehicle may take within the environment, "virtual" boundaries such as lane markings, or elements representing logical collections or groups of other elements. The map data may also include data that characterizes or otherwise describes elements within the environment (e.g., data describing the geometry, dimensions, shape, etc. of an object), or data that describes the type, function, operation, purpose, etc. of elements within the environment (e.g., speed limits, lane restrictions, traffic device operation or logic, etc.). In some embodiments, the atlas subsystem 160 may provide map data in a format that defines the position of at least some elements in a geographic region primarily based on their relative positioning to each other rather than any absolute positioning within a global coordinate system. However, it should be understood that other map or atlas systems suitable for maintaining map data for use by an autonomous vehicle may be used in other embodiments, including systems based on absolute positioning. Additionally, it should be understood that in some embodiments, at least some of the map data generated and / or utilized by the atlas subsystem 160 may be transmitted to a remote assistance system.

[0052] It should be understood that Figure 1 The illustrated set of components for the primary vehicle control system 120 is exemplary in nature. In some embodiments, individual sensors may be omitted Figure 1Multiple sensors of the type illustrated can be used for redundancy and / or to cover different areas around the vehicle, and other types of sensors can be used. Similarly, different types and / or combinations of control subsystems can be used in other embodiments. Additionally, although subsystems 152-160 are illustrated as separate from processor 122 and memory 124, it should be understood that in some embodiments, some or all of the functionality of subsystems 152-160 can be implemented with program code instructions 126 residing in one or more memories 124 and executed by one or more processors 122, and these subsystems 152-160 can, in some instances, use the same processor and / or memory for implementation. The subsystems in some embodiments can be implemented at least in part using various dedicated circuit logics, various processors, various field programmable gate arrays (“FPGAs”), various application specific integrated circuits (“ASICs”), various real-time controllers, etc., and as described above, multiple subsystems can utilize common circuits, processors, sensors, and / or other components. Additionally, the various components in the primary vehicle control system 120 can be networked in various ways.

[0053] In some embodiments, vehicle 100 can also include an auxiliary vehicle control system 170, which can be used as a redundant or backup control system for vehicle 100. In some embodiments, auxiliary vehicle control system 170 can be capable of fully operating autonomous vehicle 100 in the event of an adverse event in primary vehicle control system 120, while in other embodiments, auxiliary vehicle control system 170 can have only limited functionality, e.g., to perform a controlled stop of vehicle 100 in response to an adverse event detected in primary vehicle control system 120. In still other embodiments, auxiliary vehicle control system 170 can be omitted.

[0054] Generally, countless different architectures including various combinations of software, hardware, circuit logic, sensors, networks, etc. can be used to implement Figure 1 the various components illustrated. Each processor can be implemented as, for example, a microprocessor, and each memory can represent a random access memory (RAM) device including a main memory, as well as any supplementary levels of memory, such as cache memory, non-volatile or backup memory (e.g., programmable or flash), read-only memory, etc. Additionally, each memory can be considered to include memory storage physically located elsewhere in vehicle 100, e.g., any cache memory in a processor, and any storage used as virtual memory, e.g., stored on a mass storage device or on another computer or controller. Figure 1 One or more of the processors illustrated or a completely separate processor can be used to implement additional functions in vehicle 100 other than for autonomous control purposes, e.g., to control an entertainment system, operate doors, lights, convenience features, etc.

[0055] In addition, for additional storage, vehicle 100 may also include one or more mass storage devices, such as, for example, a floppy disk drive or other removable disk drive, a hard disk drive, a direct access storage device (DASD), an optical disk drive (e.g., a CD drive, a DVD drive, etc.), a solid state storage drive (SSD), a network attached storage, a storage area network, and / or a tape drive, etc. In addition, vehicle 100 may include a user interface 172 to enable vehicle 100 to receive multiple inputs from a user or operator and generate outputs for the user or operator, such as, for example, one or more displays, a touch screen, a voice and / or gesture interface, buttons, and other tactile controls, etc. Otherwise, user input may be received via another computer or electronic device, such as, for example, via an application on a mobile device or via a web interface, such as from a remote operator.

[0056] In addition, vehicle 100 may include one or more network interfaces, such as network interface 174, suitable for communicating with one or more networks 176 (e.g., LAN, WAN, wireless network, and / or the Internet, etc.) to permit communicating information with other vehicles, computers, and / or electronic devices, including, for example, a central service, such as a cloud service, from which vehicle 100 receives environmental and other data for its autonomous control. In the illustrated embodiment, for example, vehicle 100 may communicate with various cloud-based remote vehicle services 178, at least for purposes of implementing the various functions described herein, including a map or mapping service or system 180, a remote assistance service or system 182, and a real-time mapping service or system 184. The map or mapping service or system 180 may be used, for example, to maintain a global repository describing one or more geographical regions of the world, and to deploy portions of the global repository to one or more autonomous vehicles, update the global repository based on information received from one or more autonomous vehicles, and otherwise manage the global repository. The remote assistance service or system 182 may be used, for example, to provide remote assistance support to vehicle 100, such as, for example, by communicating with a remote assistance subsystem 186 residing in the primary vehicle control system 120, as will be discussed in more detail below. The real-time mapping service or system 184 may be used to disseminate various observations collected by one or more autonomous vehicles to effectively supplement the global repository maintained by the map or mapping service or system 180. The terms "service" and "system" are generally used interchangeably herein and generally refer to any computer function capable of receiving data from and providing data to an autonomous vehicle. In many instances, these services or systems may be considered remote services or systems since they are typically external to and communicate with the autonomous vehicle.

[0057] Figure 1Each of the illustrated processors and the various additional controllers and subsystems disclosed herein generally operate under the control of an operating system and execute or otherwise rely on various computer software applications, components, programs, objects, modules, data structures, etc., as will be described in more detail below. Additionally, the various applications, components, programs, objects, modules, etc. may also be executed on one or more processors in another computer coupled to vehicle 100 via a network, e.g., in a distributed, cloud-based, or client-server computing environment, such that the processing required to implement the functionality of the computer program can be distributed across multiple computers and / or services via the network. Further, in some embodiments, data recorded or collected by the vehicle may be manually retrieved and uploaded to another computer or service for analysis.

[0058] Generally, routines that are executed to implement the various embodiments described herein, whether implemented as part of an operating system or as a particular application, component, program, object, module, or sequence of instructions, or even a subset thereof, will be referred to herein as "program code". Program code typically includes one or more instructions that reside at various times in various memory and storage devices and, when read and executed by one or more processors, perform the steps required to execute the steps or elements embodying aspects of the present invention. Additionally, while the present invention has been and will hereinafter be described in the context of fully functional computers and systems, it should be understood that the various embodiments described herein are capable of being distributed as a program product in a variety of forms, and the present invention applies equally regardless of the specific type of computer-readable medium used to actually effect the distribution. Examples of computer-readable media include tangible non-transitory media such as volatile and non-volatile memory devices, floppy disks and other removable disks, solid state drives, hard disk drives, magnetic tape, and optical discs (e.g., CD-ROM, DVD, etc.).

[0059] Additionally, the various program code described below may be identified based on the application in which it is implemented in a particular embodiment. However, it should be understood that any particular program terminology used below is for convenience only and, therefore, the present invention should not be limited to use in any particular application identified and / or implied by such terminology. Further, considering the typically infinite number of ways in which a computer program may be organized into routines, procedures, methods, modules, objects, etc., and the various ways in which program functionality may be distributed among the various software layers (e.g., operating system, libraries, APIs, applications, applets, etc.) residing within a typical computer, it should be understood that the present invention is not limited to the particular organization and distribution of program functionality described herein.

[0060] Those skilled in the art will recognize that Figure 1The illustrated exemplary environment is not intended to limit the present invention. In fact, those skilled in the art will recognize that other alternative hardware and / or software environments may be used without departing from the scope of the present invention.

[0061] Remote assistance system

[0062] Operating an autonomous vehicle in the complex and dynamic environment of regular vehicle operation typically requires handling various situations, which, although relatively infrequent, many autonomous vehicles still often encounter over time. Autonomously handling these infrequent situations in a high-performance manner can be challenging, and some proposed methods for addressing these infrequent situations involve using remote assistance or "human-in-the-loop" techniques, such that a human operator who may be located remotely from the vehicle can make decisions and assist in guiding the vehicle when the vehicle encounters some of these infrequent situations.

[0063] Some proposed remote assistance methods focus on direct control of the vehicle by a remote operator, whereby sensor data collected by the vehicle is provided to the operator, and the operator can directly control the vehicle from a remote location. However, it has been found that direct control of the vehicle in such cases typically requires a fast, responsive, and reliable network connection between the remote operator and the autonomous vehicle. However, the network connectivity and latency of autonomous vehicles can vary significantly based on location (e.g., urban or rural, highway or bypass, etc.) and network congestion. Additionally, even when provided with sensor data collected by the vehicle, a remote operator may still lack complete situational awareness due to the fact that they are physically not inside the vehicle.

[0064] On the other hand, many of the disclosed embodiments discussed below may focus on indirect control methods, whereby a remote assistance service or system can provide advice or recommendations to an autonomous vehicle, while any such instructions or recommendations need to be verified by the autonomous vehicle before implementation. By doing so, the performance of the vehicle can be effectively decoupled from the performance of the network that links the remote assistance service or system to the autonomous vehicle.

[0065] For example, Figure 2FIG. illustrates an example implementation of a remote assistance enabled system or service 200, where an autonomous vehicle 202 interfaces with a remote remote assistance system 204 via a network 206. The remote remote assistance system 204 may be physically remote from the autonomous vehicle 202 and will generally support interfaces with multiple vehicles to enable multiple remote assistance operators to interact with multiple vehicles simultaneously. As will become more apparent below, in some implementations, the remote assistance operator may actively and continuously monitor individual vehicles, while in other implementations, individual remote assistance operators may interact with multiple vehicles at different times, e.g., such that a particular operator may support multiple vehicles at once. In some implementations, for example, whenever certain conditions occur (e.g., various uncommon situations that may benefit from remote assistance support), the remote assistance operator may selectively couple to a particular autonomous vehicle on demand, e.g., in response to a request generated by the vehicle. In some implementations, a pool of operators may support a pool of autonomous vehicles, and the remote assistance system may initiate a remote assistance session on demand, e.g., based on requests initiated by the autonomous vehicle, the remote assistance system, or both.

[0066] In some implementations, remote assistance support may be implemented using a remote assistance control module 208 and a remote assistance camera module 210 in the autonomous vehicle 202 that communicate with a remote assistance base module 212 in the remote assistance system 204. The modules 208 and 210 of the vehicle 202 may be coupled to the network 206 via a modem 214, while the module 212 of the remote assistance system 204 may be coupled to the network 206 via a modem aggregator unit 216 capable of communicating with multiple modems 214 of multiple autonomous vehicles 202. The network 206 may be implemented in part using a wireless network such as a 4G, LTE, or 5G network, a satellite network, or some combination thereof, but the invention is not limited thereto.

[0067] In some implementations, the remote assistance control module 208 may reside within the main computing system 218 of the vehicle 202 and may interface with each of the vehicle's autonomous system 220 and platform 222 to collect data from the main computing system and stream the data to the remote assistance system 204, as well as receive and process operator input received from the remote assistance system 204. In some implementations, the main computing system 218 may be implemented in a manner similar to Figure 1 the illustrated main vehicle control system 120, where the autonomous system 220 represents a high-level autonomous control subsystem, such as positioning, planning, perception, etc., and where the platform 222 represents low-level vehicle control such as provided by the control subsystem 158. However, it should be understood that in other implementations, the remote assistance control module 208 may interact with any autonomous or control-related aspect of the vehicle 202.

[0068] In some embodiments, the remote assistance camera module 210 may reside within a camera system 224 that manages on-vehicle cameras on the managed vehicle 202, and in some embodiments, module 210 may stream camera feed data streams collected from the on-vehicle cameras to the remote assistance system 204 for viewing by an operator during a remote assistance session. In some embodiments, module 210 may dynamically change the data streamed from the on-vehicle cameras, e.g., to change the priority, quality, and / or resolution of each camera feed.

[0069] Although modules 208 and 210 are implemented separately in Figure 2 , in other embodiments, the functionality assigned to each module may be changed, or the functionality may be combined into a single module or split into more than two modules. Accordingly, the present invention is not limited to Figure 2 the particular architecture shown.

[0070] The remote assistance base module 212 communicates with modules 208 and 210 during a remote assistance session with the vehicle 202 and may further manage multiple vehicles and multiple sessions with multiple operators. In some embodiments, module 212 may also manage the scheduling, initiation, and termination of sessions.

[0071] The remote assistance operator user interface 226 is coupled to module 212 to provide a user interface through which an operator (e.g., a human operator) may communicate with the vehicle 202 during a session. The user interface may be implemented in any number of suitable ways and may utilize text, graphics, video, audio, virtual reality or augmented reality, keyboard input, mouse input, touch input, voice input, gesture input, etc. Dedicated or custom controls and / or indicators may also be used in some embodiments. Additionally, in some embodiments, an application that may execute, for example, on a desktop computer or laptop computer, a mobile device, etc., may be utilized to interact with the operator, while in other embodiments, a web-based or remote interface may be used. In one example embodiment discussed in more detail below, for example, interface 226 may be a web-based interface that interacts with the operator via a touch screen display.

[0072] The remote assistance system 204 may also include one or more autonomous components 228 that interface with module 212. The autonomous components 228 may include functionality that replicates similar components in the vehicle 202 and / or various components that the vehicle may also access for use in conjunction with the vehicle's main control (e.g., as described below in connection with Figure 3The components 240, 242, and 244 discussed). For example, in some embodiments, the module 212 can access the same map data utilized by each vehicle, such as the map data provided by the atlas system as described above, as well as a similar layout function used by each vehicle to layout the map data in the vicinity of the vehicle. By doing so, the module 212 can effectively reconstruct a digital map of at least the static objects in the vicinity around the vehicle without having to receive the entire digital map from the vehicle itself, thereby reducing the amount of data streamed by the vehicle for the purpose of reconstructing the environment around the vehicle. In some embodiments, the vehicle can provide the current pose of the vehicle as well as data regarding any dynamic entities detected by the perception system (e.g., other vehicles, pedestrians, or other actors or objects detected in the environment but not represented in the map data), and based on this more limited amount of data, a graphical depiction of the immediate vicinity around the vehicle can be generated to be shown to a remote operator. In some embodiments, the autonomous component can also replicate the functions implemented in the vehicle 202 to enable a local evaluation of how the vehicle would respond to certain instructions from the remote assistance system, and in some embodiments, the autonomous component can have functions similar to those implemented in the vehicle 202 but with greater capabilities and / or access to greater computational resources than the capabilities available in the vehicle.

[0073] In addition, in some embodiments, the remote assistance system can also be autonomous in nature, whereby the remote assistance system is effectively the remote assistance operator with which the autonomous vehicle interacts during a remote assistance session. In such an instance, the remote assistance system can evaluate the current situation of the autonomous vehicle and send commands, requests, instructions, suggestions, etc. to address any situation that triggered the remote assistance session. In some embodiments, for example, the remote assistance system can access more computational power than the on-vehicle provided computational capabilities of the autonomous vehicle, and thus the remote assistance system is capable of performing computationally complex evaluations to assist the autonomous vehicle.

[0074] The remote assistance system 204 may also include an operations / fleet interface 230 to facilitate communication with other services and / or systems that support autonomous vehicles. For example, in some embodiments, it may be desirable to provide the ability to request roadside assistance or recovery of an autonomous vehicle, or to provide log data for diagnosing vehicle problems. It may also be desirable to propagate data collected during a remote operation session (e.g., data related to lane closures, detected construction or accidents, etc.) to other vehicles in the fleet. Additionally, data received and / or generated by the remote assistance system can be used as training data for various components of an autonomous vehicle, e.g., to improve the performance of detectors and reduce the occurrence of false positives, or to improve the scenario selection and other decisions made by the autonomous vehicle in response to certain sensor inputs. In other embodiments, other external services and / or systems may also interface with the remote assistance system, which will be apparent to those of ordinary skill in the art who benefit from this disclosure.

[0075] Turning now to Figure 3 , an example embodiment of the remote assistance control module 208 and the various interfaces it supports is illustrated in more detail. In some embodiments, for example, module 208 may include interfaces, e.g., (via modem 212) to the remote assistance system 204, to the platform 222, and application programming interfaces (APIs) to each of the sensing component 240, the map projection component 242, and the planner component 244 of the autonomous component 220.

[0076] For the interface with the remote assistance system 204, the remote assistance control module 208 may be configured to transmit autonomous data (e.g., map data, sensing data, route data, planning data), sensor data, telemetry data, etc. to the remote assistance system 204 to provide the current state of the vehicle to the remote assistance system. The remote assistance control module 208 may also be configured to transmit various remote assistance requests to the remote assistance system 204 and receive various remote assistance commands therefrom. Additionally, the remote assistance control module 208 may be configured to receive visualization requests and transmit the requested visualizations to the remote assistance system 204.

[0077] For the interface with the platform 222, the remote assistance control module 208 may be configured to receive vehicle state information (e.g., various types of diagnostic and / or sensor data) from the platform and issue various low-level commands to the platform, e.g., honk the horn, activate or deactivate hazard lights, change gears, disable the vehicle, initiate a controlled stop, etc.

[0078] For the interface with the sensing component 240, the remote assistance control module 208 may be configured to receive from the sensing component the actors and / or the traces of the actors detected in the environment, the detections of the various detectors 246 implemented in the sensing component, and other sensing-related data. The module 208 may transmit all such data as autonomous data to the remote assistance system.

[0079] For the interface with the map layout component 242, the remote assistance control module 208 may receive from the map layout component 242, for example, local map data, route data, and other map-related data. In some instances, the module 208 may also transmit map patches to the map layout component 242, for example, to generate lane closures, traffic device coverage, new destinations, virtual path suggestions, etc., or to clear previously generated map patches applied to the local map stored in the component 242 when a previous lane closure has been removed. In some instances, the map layout component 242 may also forward map and route updates to the planner component 244 to update the scenarios envisioned by the planner component during the operation of the autonomous vehicle.

[0080] For the interface with the planner component 244, the remote assistance control module 208 may receive, for example, the generated plans, actor attributes, alternative scenarios, and other planning-related data. The remote assistance control module 208 may also forward various remote assistance commands to the planner component 244, and receive remote assistance requests and / or feedback on the remote assistance commands from the planner component 244.

[0081] Those of ordinary skill in the art who benefit from this disclosure will understand other functions and variations. Therefore, the present invention is not limited to the specific remote assistance system embodiments discussed herein.

[0082] Remote real-time map system for autonomous vehicles

[0083] It should be understood that an autonomous vehicle can desirably know about objects and elements in its environment far enough in advance to have sufficient time to react to them. This is particularly important for stationary objects in the lane in which the autonomous vehicle is traveling, such as construction elements (e.g., cones and barrels) that define a construction zone and / or block a portion of the road. Although the sensors and perception systems of an autonomous vehicle may be able to detect such objects and elements, in many instances, the detection range is such that some objects will not be detected as early as desired to navigate properly through a construction area. In some instances, road signs warning of an upcoming construction zone can be detected and used to provide an earlier notification of potential lane closures; however, even when road signs are present, the sensors of the vehicle may be blocked by surrounding vehicles and may not provide sufficient advance notice to the autonomous vehicle. As an example, an autonomous vehicle operating in traffic on a road following and driving next to relatively tall commercial trucks and vans may have little visibility of road signs or the road ahead. Additionally, although a remote assistance system may potentially be used to navigate an autonomous vehicle through a construction zone, if the autonomous vehicle does not have sufficient notice of the presence of the construction zone (e.g., due to road signs being blocked by other vehicles), there may not be sufficient time to establish a remote assistance session before encountering the construction zone.

[0084] For example, offline map data maintained in a global repository and used to represent static objects or elements in the environment can include some construction-related information about the environment. Particularly for long-term construction projects, lane closures or changes may be of a duration of weeks or months and can thus be incorporated into the offline map data used by autonomous vehicles operating in the environment. Nevertheless, the update cycle of such map data is typically a heavyweight offline process that can take days to complete, and thus any temporary construction zone or any change to a long-term construction zone (e.g., which lanes are closed) may not be immediately reflected in the offline map data used by the autonomous vehicle. The difference between the offline map data and what the autonomous vehicle perceives in real-time can pose navigation and motion planning challenges.

[0085] However, in some embodiments of the present invention, a real-time map service or system can be used to supplement the offline map data used by an autonomous vehicle with various observations collected by various autonomous vehicles operating in the same environment. In particular, in some embodiments, the real-time map system can be configured to receive observation data from one or more autonomous vehicles operating in the environment, hold the observation data in a data store, and then propagate the observation data to one or more autonomous vehicles operating in the environment, thereby making the observation data generated by some autonomous vehicles in the environment available to other autonomous vehicles for controlling those other autonomous vehicles.

[0086] In this regard, observation data can be considered to include various types of data associated with one or more observations collected from the environment and usable by an autonomous vehicle control system in connection with controlling its autonomous vehicle. In some instances, the observation data can be associated with perception observations, which can be considered to include observations sensed by one or more sensors of the autonomous vehicle and / or derived from data sensed by such sensors, such as, for example, sensor data such as image data, LIDAR data, telemetry data, etc., and the traces, paths, and / or locations of objects or elements sensed in the environment including, but not limited to, static or stationary objects or elements, dynamic or moving objects or elements, physical objects or elements, logical objects or elements, etc. In some embodiments, for example, the observation data based on perception observations can include the traces of signs and / or construction elements detected in the environment (e.g., cones, barrels, Jersey barriers, etc.), the traces of other vehicles, pedestrians, and / or other moving objects within the environment, the physical and / or logical boundaries of the road (e.g., defining lanes, shoulders, etc.), and blocked areas representing non-drivable regions. In some embodiments, the blocked areas can be used to define traffic-closed regions of a digital map and, in some instances, can be used, for example, to represent multiple associated construction elements detected in the environment, thus obviating the need to individually track all construction elements. The observation data can also include data associated with accident scenes, such as, for example, police cars, cones, or other construction elements temporarily placed in the road to block the accident scene, the blocked areas thus defined, the traces of other vehicles navigating around the accident scene, etc.

[0087] The observation data can also be associated with operational observations based on decisions made by the autonomous vehicle when operating within the environment and / or operations performed by the autonomous vehicle and / or performance evaluations of such decisions and / or operations (e.g., whether a decision is determined to be a good decision or a bad decision, whether the determined motion plan is provided for smooth and efficient operation of the autonomous vehicle, etc.). In some embodiments, for example, the observation data based on operational observations can include the determined motion plan and its relative performance, and, in some instances, the identified optimal and / or suboptimal actions. Optimal actions can include, for example, actions or operations determined to provide the desired performance of the autonomous vehicle in response to a particular situation, while suboptimal actions can include actions or operations taken by the autonomous vehicle that are determined to provide suboptimal performance, such as, for example, where the motion plan requires sudden corrections in direction and / or speed, requires the autonomous vehicle to stop or pull over, and / or subjects the autonomous vehicle and its passengers to undesired forces (e.g., driving over bumps too fast, making sharp turns, decelerating quickly, etc.).

[0088] Thus, contrary to “offline” map systems (such as atlas systems) that typically provide relatively stable representations of static objects or elements, including physical objects or elements such as road boundaries, buildings, and logical objects such as lanes, intersections, traffic signals, etc., a real-time map system or service can be considered to provide an “online” map system that provides relatively current observational data collected by other autonomous vehicles, sometimes in real-time or near real-time. In the illustrated embodiments, a real-time map system can be distinguished from an offline map system based on the currency of the information (e.g., minutes or hours relative to days or weeks), based on the manner in which it is propagated to the autonomous vehicle (e.g., based on the current location of the autonomous vehicle and requested on-demand by the autonomous vehicle or pushed by the real-time map system relative to being published to a fleet of autonomous vehicles via versioned updates), and / or based on the degree of verification and / or quality assurance (e.g., little or no verification and / or quality assurance in the real-time map system relative to comprehensive verification and quality assurance prior to being propagated as an offline map update). Additionally, in some embodiments, a real-time map system can be distinguished from an offline map system based on the fact that the observational data propagated by the real-time map system can be based on the observations of individual autonomous vehicles, while the map data maintained in an offline map system can be based primarily on aggregated observations collected and / or verified by multiple autonomous vehicles. It should be noted that for the purposes of this disclosure, and unless otherwise explicitly stated, the terms “object” and “element” can be considered synonymous with each other.

[0089] In the context of the present disclosure, map data provided by an offline map system such as an atlas system (which can be referred to as offline map data) can be used to generate a digital map representing the environment in which an autonomous vehicle operates. In some embodiments, such a digital map can be referred to as a road area layout. While in some embodiments, the offline map data can be provided by a remote offline map system when the autonomous vehicle is operating in the environment, in other embodiments, the offline map data is stored in the autonomous vehicle and is locally available during operation, where updates to the offline map data are provided, for example, periodically via versioned updates published to the fleet.

[0090] Furthermore, the observational data provided by a real-time map system can be used to enhance the offline map data used to generate a digital map to generate an enhanced digital map, where the observational data is effectively “fused” with the offline map data, e.g., by supplementing the offline map data (e.g., by adding blocked areas, boundaries, construction elements, signs, etc.) and / or replacing or overriding the offline map data (e.g., by closing off normally drivable areas of traffic and / or opening previously non-drivable areas). In some embodiments, in the case where the digital map is implemented as a road area layout, the observational data can be used to generate an enhanced road area layout.

[0091] Furthermore, as will be understood from the discussion below, such enhancement of the digital map can be implemented using map fusion or similar functional components of an autonomous vehicle control system that fuse additional information with the digital map or enhanced digital map, including, for example, perception observations generated from sensor data of the autonomous vehicle, such as detected signs, construction elements, and traces of other objects and elements detected in the environment, as well as various perceived lane boundaries detected in the environment. Additionally, in some embodiments, the map fusion or similar functional components can further fuse information received from a remote teleassistance system, such as a suggested path, blocked areas, etc.

[0092] While the enhanced digital map may be useful for other operations in conjunction with the operation of an autonomous vehicle, the disclosure below primarily focuses on using such enhanced digital map in performing motion planning, which can be considered to include any function or algorithm suitable for generating a path of motion of the autonomous vehicle within a relatively short time frame (as opposed to route planning, which is more concerned with navigating to a specific geographical destination via a route and not with the actual path taken by the autonomous vehicle within the road it travels while following the route).

[0093] In some embodiments, for example, the observation data can include the paths of one or more vehicles (including autonomous and / or non-autonomous vehicles) while navigating through an area, which can allow the motion planner to select a motion path for the autonomous vehicle based on these other vehicle paths. Thus, for example, if multiple non-autonomous vehicles follow a specific path through a rocky construction area (e.g., changing lanes or even changing to the other side of a divided highway), the motion planner can use these vehicle paths to generate a similar motion path for the autonomous vehicle. Similarly, for any suboptimal actions represented in the observation data, the motion planner can use these actions to reject potential motion paths that would result in these suboptimal actions. Thus, in the case where another autonomous vehicle follows a specific motion path through a rocky construction area that causes a sudden correction of the direction and / or speed of the autonomous vehicle, that motion path can be provided to other autonomous vehicles so that those other autonomous vehicles can reject similar motion paths generated via motion planning.

[0094] Furthermore, map fusion or similar functional components can also be used to provide the observation data collected by the autonomous vehicle to a real-time map system for dissemination to other autonomous vehicles operating in the environment.

[0095] As will be discussed in more detail, in some embodiments, the real-time map service can also be used to disseminate remote assistance-related information to autonomous vehicles operating in the area. In some embodiments, such information can include location-based remote assistance triggers, which can be considered to include remote assistance information associated with specific location-based criteria that, when met, can be used to automatically trigger a remote assistance session and / or provide guidance generated by remote assistance to the autonomous vehicle. In some embodiments, for example, a location-based remote assistance trigger can include a session recommendation that recommends initiating a remote assistance session whenever an autonomous vehicle approaches a predetermined location, enabling a remote assistance operator to provide guidance to the autonomous vehicle, e.g., in areas or scenarios where remote assistance is known to be useful - such as construction areas, checkpoints, weigh stations, warehouse entrances / exits, or any other complex areas encountered by autonomous vehicles.

[0096] In addition, in some embodiments, when an autonomous vehicle is operating in an area associated with location-based criteria, the location-based remote assistance trigger can include recommended actions, e.g., a recommended follow path, a recommended drive-in lane, or any other action that can be recommended by a remote assistance operator. In some embodiments, the recommended actions can also be presented during a remote assistance session. While in some instances both the session recommendation and the recommended actions can be associated with the location-based remote assistance trigger, in other instances, the recommended actions can be used to potentially obviate the need to initiate a remote assistance session, enabling the autonomous vehicle to effectively navigate through a potentially problematic area without the assistance of a remote assistance operator, but still using the recommendations made by a remote assistance operator when assisting another autonomous vehicle that has previously navigated through the area.

[0097] In some embodiments, the location-based remote assistance trigger can also be associated with observation data collected by another autonomous vehicle in conjunction with a remote assistance session conducted with that other autonomous vehicle, such that the observation data can be used by other autonomous vehicles. For example, the vehicle path of the autonomous vehicle participating in the remote assistance session and / or the vehicle paths of other vehicles sensed by the autonomous vehicle can be provided for use in association with the location-based remote assistance trigger. Similarly, any optimal and / or suboptimal actions performed by the autonomous vehicle during the remote assistance session can be propagated to other autonomous vehicles via the location-based remote assistance trigger. Any other observation data collected in association with the remote assistance session can also be propagated to other autonomous vehicles via the location-based remote assistance trigger, e.g., to assist the autonomous vehicle in controlling its movement and / or to assist another remote assistance operator when initiating a new remote assistance session with another autonomous vehicle.

[0098] Location-based criteria used to trigger location-based remote assistance can vary in different embodiments. While in some embodiments, the criteria may only specify a location at which the autonomous vehicle triggers when within a predetermined distance of that location, in other embodiments, more detailed criteria may be used, e.g., by triggering a geofence whenever the autonomous vehicle enters a defined geofence, and / or by specifying the route, road, lane, direction, etc. of the autonomous vehicle (e.g., to avoid triggering when the autonomous vehicle is traveling in the opposite direction on a separate road), etc.

[0099] It should also be understood that in some embodiments, e.g., assuming that a construction area can be removed or can change (e.g., switching a lane closure from the left lane to the right lane), the observed data maintained in the real-time map system can be transient or temporary in nature. Thus, in some embodiments, it may be desirable to associate the observed data with an expiration time or duration such that the observed data can be automatically removed from the real-time map system if no similar observations are collected by other autonomous vehicles for a period of time. It may also be desirable to utilize an offline process to verify the observed data and / or potentially perform quality assurance on the observed data with the help of a human operator to remove unreliable observed data and / or potentially propagate reliable observed data to an offline map system for inclusion in a global repository.

[0100] Those of ordinary skill in the art who benefit from this disclosure will understand other variations.

[0101] Figure 4 An example system 250 suitable for leveraging a real-time map system and a remote assistance system to facilitate the control of an autonomous vehicle is illustrated. In this embodiment, a map fusion component 252 integrates or fuses various static elements mapped by an offline map system in the environment around the autonomous vehicle (e.g., provided in the form of a road area layout by a road area generator 254) with various observations currently detected in the environment by a perception component 256 to generate a digital map that can be consumed by a motion planner component 258 to generate a path for the autonomous vehicle to follow. The digital map can contain mapped static elements and perception elements from the offline map system, such as traces of detected but unmapped signs, unmapped construction elements, and other unmapped objects and elements detected in the environment, as well as various perceived lane boundaries detected in the environment. Thus, the observed data from the perception component 256 can be used to supplement the stored map data with additional elements detected in the environment and / or changes to existing mapped elements, thereby providing a more current representation of the environment around the autonomous vehicle.

[0102] In addition, the real-time map system 258 also provides the map fusion component 252 with observation data associated with observations collected by other autonomous vehicles, in order to enhance the digital map with these other observations. In this way, the enhanced digital map provided to the motion planner component 258 can include offline map data as well as observation data locally collected by autonomous vehicles (from the perception component 256) and by other autonomous vehicles operating in the same environment (from the real-time map system 258).

[0103] In addition, the real-time map system 258 can also provide the map fusion component 252 with location-based remote assistance triggers, for example, to trigger a remote assistance session with the remote remote assistance system 262 at a distance. For example, in response to meeting the location-based criteria of the location-based remote assistance trigger, a request for a remote assistance session (block 264) can be forwarded to the remote assistance system 262, where the request is forwarded through the interface 266. Then, the map fusion component 252 can receive the feedback generated by the remote assistance operator during the remote assistance session through the interface 268 and incorporate it to assist in operating the autonomous vehicle (block 270). The interfaces 266, 268 (which can be implemented in the same component in some embodiments) can be used to transfer data - for example, situation awareness information - from the map fusion component 252 to the remote remote assistance system 262 at a distance, and to transfer data - for example, operator suggestions, commands, or requests for additional information - from the remote remote assistance system 262 at a distance back to the map fusion component 252.

[0104] For example, in response to the location-based remote assistance trigger that triggers the remote assistance request in block 264, the remote remote assistance system 262 at a distance can obtain or create a remote assistance session (block 274) and provide situation awareness information to the remote assistance operator (block 276). The situation awareness information or data can include any information or data that may be relevant to the current situation of the autonomous vehicle and / or useful to the remote assistance operator during the remote assistance session, such as the current path, the paths of other vehicles, map data of the area around the autonomous vehicle, telemetry data, camera feed data, and / or any other data that may be relevant to evaluating the current situation encountered by the autonomous vehicle. Once the situation data is provided, the remote remote assistance system at a distance waits for the operator to evaluate the situation and provide operator input (block 278), which is then incorporated into block 270 as described above. After providing the operator input to the autonomous vehicle, the operator can continue to monitor the operation of the autonomous vehicle and / or provide additional input (block 280) until the remote assistance interaction is completed, at which time the operator can close the session (block 282).

[0105] Figure 5Next, an example real-time map and a remote assistance enabling system 300 incorporating an autonomous vehicle 302 communicating with a set of online services 304 are illustrated. The motion planner component 306 receives data, such as an enhanced road area layout, from the map fusion component 308. Similar to Figure 4 the map fusion component 252, the map fusion component 308 receives a digital map, such as a road area layout, from the road area generator component 310. The component 310 can generate a road area layout by accessing an on-vehicle map 312 to retrieve offline map data suitable for generating a digital map of the environment around the autonomous vehicle 302 during operation. The map fusion component 308 also receives observation data from the perception component 314, which collects observations using one or more sensors in the manner described above. In some embodiments, the perception component may also directly provide the motion planner component 306 with the tracks of various dynamic objects. The map fusion component 308 may also receive remote assistance data, such as remote assistance operator input, from the remote assistance component 316 communicating with the distal remote assistance system 318, and additional observation data from the real-time map system 320, such that the data collected from the components 310, 314, 316, and 320 can be fused into the enhanced road area layout used by the motion planner component 306 when generating the motion path of the autonomous vehicle.

[0106] The observation data provided by the real-time map system 320 is stored in the real-time map database or data store 322 and includes observation data collected from one or more other autonomous vehicles 324. The real-time map system 320 typically communicates bidirectionally with the autonomous vehicle 302 and other autonomous vehicles 324 to collect the observation data of the autonomous vehicles 302, 324 and propagate the observation data collected by the autonomous vehicles 302, 324 operating in a specific part or area of the environment to other autonomous vehicles 302, 324 operating in the same part or area of the environment. In addition, the remote assistance system 318 may also communicate bidirectionally with the real-time map database 322 via the real-time map system 320 to enable the remote assistance operator to store remote assistance data, such as location-based remote assistance triggers, to be propagated to the autonomous vehicle via the real-time map system 302, and to retrieve observation data from the real-time map database 322 in conjunction with a remote assistance session with the autonomous vehicle 302. As described above, in some embodiments, the remote assistance system 318 may also communicate bidirectionally with the autonomous vehicle 302 (e.g., via the remote assistance component 316) to allow the autonomous vehicle to provide situational awareness data to the remote assistance operator and to allow the remote assistance operator to provide suggested actions to the autonomous vehicle. In some embodiments, there may be no direct link between the remote assistance system 318 and the real-time map system 320, such that the communication between these components can be processed by the map fusion component 308 and the remote assistance component 316.

[0107] Similarly as described above, as opposed to the offline map service 326 that provides offline map data to the autonomous vehicle fleet, such as a map system or service, the real-time map system 320 and the real-time map database 322 represent an online map system that disseminates observation data within the autonomous vehicle fleet. The offline map service 326 may include, for example, an offline map database 328 that maintains a global repository of offline map data. The map publishing component 330 can be used to generate versioned updates of the offline map database, and the map deployment component 332 can be used to deploy or disseminate the database updates to the autonomous vehicle fleet. It should be understood that the size of the global repository of offline map data may be large, so in some embodiments, only a portion of the offline map data corresponding to a specific region (e.g., the city, state, county, etc. in which the autonomous vehicle 302 operates) can be deployed to the autonomous vehicle and maintained in its on-vehicle map 312. Additionally, based on the movement of the autonomous vehicle into adjacent regions, additional portions of the offline map data can be transmitted to the autonomous vehicle by the service 326, such that the on-vehicle map 312 includes sufficient offline map data to operate the autonomous vehicle at its current location.

[0108] In some embodiments, the real-time map system 320 and the offline map service 326 can be completely independent of each other, while in other embodiments, and as Figure 6 illustrated by the system 300’ of, the map operator component 334 of the offline map service 326’ can communicate with the real-time map system 320’ such that the observation data collected by the real-time map system 320’ can be incorporated into the offline map database 328. In such a method, frequently changing observations (such as construction elements) can be maintained in both the real-time map database 322 and the offline map database 328 (potentially in a separate layer with lower accuracy requirements than other offline map data), and such observations can be verified (e.g., by the map operator component 334, optionally under the guidance of a human operator) before being published, and then given priority when they are loaded on the autonomous vehicle and are found to overlap with unvalidated observation data from the real-time map database 322. Additionally, such offline map data can utilize an automatic expiration policy similar to the observation data in the real-time map database 322 to automatically expire observations that no longer exist in the real world.

[0109] As will be understood by those of ordinary skill in the art who benefit from this disclosure, other ways of partially or fully integrating the real-time map system with the offline map system can be used in other embodiments.

[0110] Now turning to Figure 7, which illustrates an example autonomous vehicle 350 including an autonomous vehicle control system 352 that interfaces with both a real-time map system and a remote assistance system for autonomous operation of the autonomous vehicle 350. It should be understood that in other embodiments, the autonomous vehicle control system may interface with only the remote assistance system or only the real-time map system, and thus the present invention is not limited to the specific embodiments discussed herein. Additionally, Figure 7 only components of the autonomous vehicle control system 352 related to motion and route planning are illustrated, and thus for simplicity purposes, other components not related to these functions are omitted from the figure.

[0111] One or more image sensors or cameras 354 and one or more LIDAR sensors 356 operate as sensors for use in conjunction with motion planning. The cameras 354 output camera data to the perception component 358, and specifically to an object classifier component 360 that also receives the output of the LIDAR sensors 356. The object classifier component 360 can be used to identify physical objects in the environment surrounding the autonomous vehicle. While component 360 can detect a wide variety of physical objects, from the perspective of the functions described herein, two specific types of objects (construction elements and signs) are shown as being output by the object classifier component 360. However, it should be understood that the types of objects that can be detected are numerous, and in other embodiments, the perception component can be configured to detect many other types of objects, and thus the output of the construction elements and signs as Figure 7 illustrated is for illustrative purposes only.

[0112] The perception component 358 may also include a sign classifier component 362 that receives objects detected by the object classifier component 360 as signs and determines the logical significance of such signs, such as the type of sign (speed limit, warning, road closure, construction notice, etc.), whether the sign is advisory or mandatory, the location and / or lane affected by the sign (e.g., right lane closed 2 miles ahead), etc. In other embodiments, the functions of components 360, 362 may be combined, or additional purpose-specific classifier components may be used.

[0113] The objects detected by the perception component 358 are provided to a tracking component 364 that maintains a track of each detected object (as well as other stationary and / or moving objects detected in the environment) over time. It should be understood that since the autonomous vehicle is typically in motion, a track can still be used to represent a stationary object because its relative position to the autonomous vehicle changes over time due to the movement of the autonomous vehicle itself.

[0114] The LIDAR sensor 356 can also provide an output to other components in the autonomous vehicle control system 352. For example, the output can also be provided to an online localization component 366 that determines the current position and orientation of the autonomous vehicle. The global pose component 368 can receive the output of the LIDAR sensor 356 to determine the global pose of the autonomous vehicle, which is output to the pose filter component 370 and fed back as an additional input to the global pose component 368. The global pose can also be output to the lane alignment component 372, which receives the output of the LIDAR sensor 356 and the output of the lane boundary component 374 as inputs, and provides another input to the pose filter 370. Although in some embodiments, the global pose may not be an input to the lane alignment component 372. In some embodiments, lane alignment can be an input for determining the global pose.

[0115] The lane boundary component 374 receives the output of the camera 354 and / or the LIDAR sensor 356, and can be used to generate, for example, sensed lane boundaries based on the output of the camera 354 and / or the LIDAR sensor 356. These lane boundaries can then be provided to the lane alignment component 372 to refine the global pose based on the lane boundaries sensed in the environment. In some embodiments, separate camera-based lane alignment models and LIDAR-based lane alignment models can be used, with the former having a greater range but lower accuracy, and the latter having greater accuracy but a shorter range. In some embodiments, the camera-based model can be used for purposes such as detecting map errors, and the LIDER-based model can be used for lane alignment and global pose determination.

[0116] The map fusion component 376 receives the tracks output by the tracking component 364 and the sensed lane boundaries from the lane boundary component 374 as inputs, which together can be considered to represent at least a portion of the sensed observation data collected by the sensors of the autonomous vehicle. In addition, the road region generator component 378 retrieves map data from the on-vehicle map 380 and generates a baseline road region layout (RRL) of a digital map representing the regional road area around the autonomous vehicle. The map fusion component 376 fuses these inputs together to generate an augmented road region layout (ARRL), which is provided to the motion planner component 382, which generates the motion path of the autonomous vehicle at least partially on the augmented road region layout.

[0117] The motion planner component 382 may also receive as inputs at least a portion of the trace output by the tracking component 364 and a desired route from the route planner component 384, which is typically generated from the on-vehicle map 308 and provides high-level guidance regarding the desired route to the desired destination. Additionally, in some embodiments, the additional blocked area component 386 may receive at least a portion of the trace output by the tracking component 364 and the enhanced road area layout from the map fusion component 376 to determine one or more blocked areas in the environment, which are output as additional traces to the motion planner component 382. As described above, the blocked areas may represent portions of the environment that are considered non-drivable and, in some embodiments, may represent a collection of associated construction elements, e.g., a collection of cones, barrels, and / or Jersey barriers that may collectively be considered to represent a boundary where autonomous vehicles are not permitted to cross.

[0118] Additionally, the remote assistance component 388 may be operatively coupled to the map fusion component 376 and the motion planner component 382 to provide an interface with a distal remote assistance system ( Figure 7 not shown). The remote assistance component 388 may output, for example, observation data to the map fusion component 376 for incorporation into the enhanced road area. One non-limiting example is the speed limit, e.g., to provide the maximum speed at which an autonomous vehicle may travel in certain lanes or on roads or within a construction zone. The remote assistance component 388 may also output, for example, suggested actions to the motion planner component 382, e.g., change lanes, stop, or pull over to the side of the road before a blocked area. The remote assistance component 388 may also receive the enhanced road area layout (or any other data collected therefrom) from the map fusion component 376 to assist the distal remote assistance operator in providing assistance to the autonomous vehicle during a remote assistance session.

[0119] The online service 390, including a distal remote assistance system and a real-time map system similar to those described above in connection with Figures 5 to 6 may interface with the map fusion component 376 and the remote assistance component 388, e.g., to provide observation data and / or location-based remote assistance triggers to the map fusion component 376 and to exchange information with the remote assistance operator during a remote assistance session.

[0120] It should be understood that the architecture of the autonomous vehicle control system 352 is merely exemplary in nature and that other architectures may be used in other embodiments. For example, among other possible variations, the blocked area component 386 may alternatively be implemented within the perception component 358 such that blocked areas are managed similar to the traces of construction elements and signs. It should also be understood that some or all of the components in the autonomous vehicle control system 352 may be implemented using programming logic and / or trained machine learning models, and the implementation of these components will be well within the capabilities of a person of ordinary skill in the art benefiting from this disclosure.

[0121] Figure 8 An example embodiment of a real-time map system 400 is illustrated. The real-time map system 400 includes a real-time map database 402 and real-time map data ingestion component 404, real-time map dissemination component 406, real-time map data quality assurance component 408, real-time map training data generation component 410, and remote assistance interface component 412. The real-time map data ingestion component 404 may be used to receive observation data forwarded by one or more autonomous vehicles and ingest the observation data into the real-time map database 402. Ingestion may also include associating new observation data with previously stored observation data and assigning or modifying the expiration criteria for the observation data to control when observation data that is no longer being observed will be automatically removed from the database. The real-time map dissemination component 406 may be used to, for example, disseminate the observation data to autonomous vehicles in the fleet based on a request by an autonomous vehicle for the observation data relevant to the vehicle's current location or automatically push based on the tracked location of the autonomous vehicle. For example, the map data dissemination component 406 may disseminate the data to a map fusion component or directly to a remote assistance component.

[0122] The real-time map data quality assurance component 408 may be used to verify the observation data received from autonomous vehicles, e.g., reject incorrect or unreliable observation data, correct or modify the observation data, disseminate reliable observation data to an offline map system, etc. The component 408 may also provide an interface with a human operator, e.g., to determine when the observation data is determined to have sufficient reliability to be incorporated into the offline map database. The real-time map training data generation component 410 may be used to generate training data for training various machine learning models, including, for example, machine learning models implemented within the real-time map system 400 and / or machine learning models implemented within the autonomous vehicle control system. The remote assistance interface 412 provides an interface to a remote assistance system, e.g., to enable a remote assistance operator to generate location-based remote assistance triggers or transfer observation data between the database 402 and the remote assistance system.

[0123] Now turning to Figure 9, this figure illustrates an example operation sequence 420 that can be implemented within an autonomous vehicle control system to utilize a real-time map system and / or a remote assistance system in conjunction with the control of an autonomous vehicle. As illustrated in block 422, a map fusion component can fuse various types of data to form an enhanced road area layout. For example, in block 424, perceptual observations sensed by one or more sensors of the autonomous vehicle can be provided, while in block 426, real-time map data can be provided by the real-time map data dissemination component 406, such as observation data collected by other autonomous vehicles and / or triggered by location-based remote assistance. Additionally, in block 428, other remote assistance data provided by the remote assistance system can be provided.

[0124] Then, motion planning can be performed, for example, by a motion planner component based on the enhanced road area layout to generate a motion plan (e.g., a motion path) in block 430. The motion plan can then be used to control the autonomous vehicle in block 432. Additionally, as illustrated in block 434, in block 434, observation data collected from the autonomous vehicle, such as perceptual observation data collected from one or more sensors of the autonomous vehicle and / or the motion plan generated by the motion planner, as well as other data, can also be forwarded to the real-time map data ingestion component 404 of the real-time map system.

[0125] Figure 10 and 11 Next, the generation and triggering of location-based remote assistance triggers are illustrated separately. In particular, Figure 10 illustrates an example operation sequence 440 for generating and storing location-based remote assistance triggers in the real-time map system 400 of Figure 8 . In some embodiments, the operation sequence 440 can be performed at least in part by a remote assistance system, such as the remote assistance system 318 of Figure 6 , and interfaced with the real-time map system 400 through the remote assistance interface 412. In embodiments where at least a portion of the operation sequence is performed in the autonomous vehicle, the map fusion component can also perform at least some operations, and in some instances, can be used to trigger a remote assistance session based on real-time map data received by the autonomous vehicle. As shown in blocks 442 to 446, a remote assistance session can be initiated (block 442), and during the session, situational awareness data and remote assistance operator input can be exchanged between the autonomous vehicle and the remote assistance system (block 444) until the remote assistance session is terminated at some point (block 446). Next, in block 448, the remote assistance operator can optionally be allowed to evaluate and / or modify any remote assistance session data, for example, to determine whether any remote assistance session data should be incorporated into the real-time map database, discarded, or otherwise.

[0126] Next, in block 450, it is determined (e.g., by a remote assistance operator or, alternatively, automatically by the remote assistance system) whether a location-based remote assistance trigger should be generated for a remote assistance session. For example, it may be determined that a particular section of a road (e.g., a construction zone) is triggering remote assistance session requests from multiple autonomous vehicles, and thus it may be desirable to trigger a new remote assistance session for any vehicle approaching the same section at an earlier time point so that the remote assistance operator can have additional time to assist the autonomous vehicle. As another example, it may be determined that a particular action recommended for a particular section of a road during a remote assistance session or a particular path recommended during a remote assistance session results in optimal or even sufficient performance, such that it may be desirable to notify another vehicle approaching the same section so that the other vehicle can implement the same action or follow the same path, potentially without even having to initiate a remote assistance session that would otherwise have to be initiated without the recommendation. As yet another example, certain types of signs and / or construction elements that have been identified with a threshold confidence (by a single autonomous vehicle or multiple autonomous vehicles) can be used to identify a particular section of a road where it may be desirable to actively trigger a remote assistance session for other autonomous vehicles.

[0127] If no location-based remote assistance trigger is recommended, sequence 440 can be completed. Otherwise, control passes to block 452 to construct a remote assistance trigger, including one or more of location-based criteria, session recommendations (e.g., automatically triggering a session), recommended actions (e.g., actions proposed for the autonomous vehicle to take), and any relevant observations that may be useful to the autonomous vehicle and / or the operator of a remotely initiated remote assistance session. Relevant observations can include perception and / or operation observations and can include, for example, any optimal or suboptimal actions taken, the paths of other vehicles (autonomous and / or non-autonomous), etc.

[0128] Figure 11 Illustrated is an example operation sequence 460 for using Figures 7 to 8 an autonomous vehicle control system and a real-time map system to activate a location-based remote assistance trigger. Specifically, in block 462, the autonomous vehicle control system receives real-time map data of its current area from the real-time map system via the real-time map data dissemination component 406. In this instance, instead of or in addition to observation data, the real-time map data includes one or more location-based remote assistance triggers associated with the current area of the autonomous vehicle. As illustrated in block 464, the autonomous vehicle is then operated in part using the real-time map data, and at some point, as indicated in block 466, it is determined that the location-based criteria of a remote assistance trigger have been met based on the autonomous vehicle's location and / or path. For example, the trigger can occur based on crossing a geofence, being within a predetermined distance of a predetermined location, and on a particular road or lane, etc.

[0129] In block 468, it is determined whether a remote assistance session should be initiated, e.g., whether a positive session recommendation is associated with a location-based remote assistance trigger. If so, control passes to block 470 to automatically initiate a remote assistance session. If not, control instead passes to block 472 to operate the autonomous vehicle based on one or more suggestions associated with the remote assistance trigger (e.g., suggested actions, suggested routes, etc.). It should also be understood that in some embodiments, the trigger can include both a suggested action and a positive session recommendation, such that each of blocks 470 and 472 can be performed for a single remote assistance trigger.

[0130] Figure 12 Next, an example operation sequence 480 is illustrated for controlling an autonomous vehicle using Figures 7 to 8 an autonomous vehicle control system and a real-time map system based on other vehicle paths and / or sub-optimal actions. Specifically, in block 482, the autonomous vehicle control system receives real-time map data of its current area from the real-time map system via the real-time map data dissemination component 406. In block 484, the paths of one or more vehicles stored as observations in the real-time map data can be added to the enhanced road area layout used by the motion planner. These paths can include, for example, paths used by other autonomous vehicles when passing through a particular section of a road, which can provide the motion planner with hints on how other autonomous vehicles navigate through the same section. Instead of or in addition to the paths generated by other autonomous vehicles, these paths can also include the paths of other (including non-autonomous) vehicles sensed by the sensors of other autonomous vehicles. Thus, for example, if a particular autonomous vehicle follows one or more manually operated vehicles through a section of a road, the traces representing the paths of the vehicles tracked via the autonomous vehicle's perception system can be used to provide the motion planner with hints on how a human driver selects to navigate through the same section. As a result, the motion planner can select a motion plan that is expected to mimic the paths of one or more other vehicles based on the real-time map data.

[0131] In addition, in block 486, one or more sub-optimal actions taken by other autonomous vehicles can be accessed from the real-time map data. Thus, for example, if another autonomous vehicle has to stop, suddenly decelerate, pull over, or suddenly change direction when navigating through the same section of a road, the motion planner can select to reject any path that would result in a similar outcome for the autonomous vehicle. Thus, in block 488, the motion planner can utilize the added paths and / or sub-optimal actions to generate a suitable motion plan for the autonomous vehicle.

[0132] Figures 13 to 14 Next, an example fusion of real-time map data and a road area layout is illustrated to generate an enhanced road area layout using Figures 7 to 8 an autonomous vehicle control system and a real-time map system. For example,Figure 13 Illustrated is a road area layout 500 generated from offline map data of an example four-lane divided highway, the four-lane divided highway including a pair of northbound lanes 502, 504, a pair of southbound lanes 506, 508, shoulders 510, and a median 512. Two dividers 514, 516 are separated by an interruption 518, for example, to allow police or emergency vehicles to cross, but the interruption is not considered a drivable area for regular vehicles via law or regulation. Arrows in each of lanes 502 - 508 also represent logical map data defining the normal driving direction in each lane.

[0133] Figure 14 Illustrated is an enhanced road area layout 520 generated by fusing observed data from a real-time map system with Figure 13 the road area layout 500, for example, after establishing (a) closing lanes 502 and 504, (b) routing northbound traffic through interruption 518 and into lane 508, and (c) requiring all southbound traffic to travel in lane 506 in a construction zone. As an example, observed data such as a blocked area 522 associated with multiple construction elements 524 can be fused into the enhanced road area layout 520. In some instances, the blocked area 522 and / or the construction elements 524 can be sensed observations from a controlled autonomous vehicle when the autonomous vehicle travels through the construction zone, or alternatively, can be observed data collected by other autonomous vehicles and propagated through the real-time map system. However, it should be understood that when a controlled autonomous vehicle travels through the construction zone, other observed data of construction element 526 may be un-sensed, and thus can represent observed data collected from other autonomous vehicles.

[0134] In addition, the fused observed data can be used to overlay the offline map data, for example, whereby the interruption 518 is effectively open to traffic, lanes 502 and 504 are closed to traffic, and the driving direction of lane 508 is reversed.

[0135] In addition, the observed data can include paths 528 of other autonomous and / or non-autonomous vehicles that have traveled through the construction zone, which can be used by a motion planner to select a motion path similar to the motion paths of other vehicles. In addition, the observed data can also include sub-optimal maneuvers, for example, as represented by path 530, which requires a sudden change in direction and a sudden decrease in speed (as represented at 532) when traveling through the construction area. The motion planner can use the sub-optimal maneuvers to select a motion path that does not replicate the undesirable results of the sub-optimal maneuvers.

[0136] It should be understood that although certain features may be discussed herein in connection with certain embodiments and / or in connection with certain figures, these features can generally be incorporated into any of the embodiments discussed and illustrated herein unless explicitly stated to the contrary. Moreover, features disclosed as combined in some embodiments can generally be implemented separately in other embodiments, and features disclosed as implemented separately in some embodiments can be combined in other embodiments, so the fact that a particular feature is discussed in the context of one embodiment rather than another should not be construed as admitting that the two embodiments are mutually exclusive. Other variations will be obvious to those of ordinary skill in the art. Accordingly, the invention resides in the appended claims.

Claims

1. An autonomous vehicle control system, comprising: One or more processors; And A memory storing instructions that, when executed by the one or more processors, cause the autonomous vehicle control system to: Receive a digital map of a portion of an environment in which an autonomous vehicle operates, the digital map defining a plurality of static elements within the portion of the environment; Receive observation data associated with one or more observations collected from the environment from a remote real-time map system; Enhance the digital map with the observation data to generate an enhanced digital map; And Use the enhanced digital map to control the autonomous vehicle.

2. The autonomous vehicle control system according to claim 1, wherein, The one or more processors are configured to use the enhanced digital map to control the autonomous vehicle by: Using the enhanced digital map to determine a trajectory of the autonomous vehicle; and Controlling the autonomous vehicle according to the trajectory.

3. The autonomous vehicle control system according to claim 1, wherein, The observation data is first observation data, and wherein, The one or more processors are further configured to: Receive second observation data collected using one or more sensors of the autonomous vehicle; and Enhance the digital map with the second observation data such that the enhanced digital map is enhanced using the first observation data and the second observation data.

4. The autonomous vehicle control system according to claim 3, wherein, The one or more processors are further configured to: Transmit the second observation data to the remote real-time map system for controlling one or more other autonomous vehicles operating in the environment.

5. The autonomous vehicle control system according to claim 1, wherein, The one or more observations collected from the environment are collected by one or more other autonomous vehicles operating in the environment and transmitted to the remote real-time map system.

6. The autonomous vehicle control system according to claim 1, wherein, The observation data defines construction elements detected in the environment.

7. The autonomous vehicle control system according to claim 1, wherein, The observation data defines a blocked area in the environment, the blocked area restricting vehicle movement within the environment.

8. The autonomous vehicle control system according to claim 1, wherein, The observation data defines one or more vehicle paths, each vehicle path associated with a detected path of a different vehicle when traveling through the environment, and wherein, The one or more processors are configured to: use the enhanced digital map to control the autonomous vehicle by determining a movement path for the autonomous vehicle using the one or more vehicle paths.

9. The autonomous vehicle control system according to claim 1, wherein, The observation data defines suboptimal actions performed by another autonomous vehicle when traveling through the environment, and wherein, The one or more processors are configured to: use the enhanced digital map to control the autonomous vehicle by determining a movement path for the autonomous vehicle using the suboptimal actions.

10. The autonomous vehicle control system according to claim 1, wherein, The one or more processors are further configured to: Receive a location-based remote assistance trigger from the remote real-time map system; And Determine the activation of the location-based remote assistance trigger at least in part based on location-based criteria associated with the location-based remote assistance trigger.

11. The autonomous vehicle control system according to claim 10, wherein, The one or more processors are further configured to: In response to determining the activation of the location-based remote assistance trigger, automatically initiate a remote assistance session with a remote assistance system at a distal end.

12. The autonomous vehicle control system according to claim 10, wherein, The one or more processors are further configured to: In response to determining the activation of the location-based remote assistance trigger, control the autonomous vehicle at least in part based on a recommended action associated with the location-based remote assistance trigger.

13. The autonomous vehicle according to claim 12, wherein, The one or more processors are configured to: Control the autonomous vehicle at least in part based on the recommended action associated with the location-based remote assistance trigger without initiating a remote assistance session with a remote assistance system at a distal end.

14. The autonomous vehicle according to claim 13, wherein, The recommended action is received from the distal real-time map system and is generated from different remote assistance sessions with different autonomous vehicles.

15. A method of operating an autonomous vehicle with an autonomous vehicle control system, comprising: Receiving a digital map of a portion of an environment in which the autonomous vehicle operates, the digital map defining a plurality of static elements within the portion of the environment; Receiving, from a distal real-time map system, observation data associated with one or more observations collected from the environment; Enhancing the digital map with the observation data to generate an enhanced digital map; And Using the enhanced digital map to control the autonomous vehicle.

16. An autonomous vehicle control system, comprising: One or more processors; And A memory storing instructions that, when executed by the one or more processors, cause the autonomous vehicle control system to: Receive a digital map of a portion of an environment in which an autonomous vehicle operates, the digital map defining a plurality of static elements within the portion of the environment; Determine observation data associated with one or more observations collected from the environment based on sensor data collected using one or more sensors of the autonomous vehicle; Enhance the digital map with the observation data to generate an enhanced digital map; Use the enhanced digital map to control the autonomous vehicle; And Transmit the observation data to a distal real-time map system for controlling one or more other autonomous vehicles operating in the environment.

17. The autonomous vehicle control system according to claim 16, wherein, The one or more processors are configured to control the autonomous vehicle using the enhanced digital map by: Using the enhanced digital map to determine a trajectory of the autonomous vehicle; and Controlling the autonomous vehicle according to the trajectory.

18. The autonomous vehicle control system according to claim 16, wherein, The observation data is first observation data, and Wherein, the one or more processors are further configured to: Receive, from the distal real-time map system, second observation data associated with one or more observations collected from the environment; and Enhance the digital map with the second observation data such that the enhanced digital map is enhanced using the first observation data and the second observation data.

19. The autonomous vehicle control system according to claim 16, wherein, the observation data defines construction elements detected in the environment using sensors of the autonomous vehicle.

20. The autonomous vehicle control system according to claim 16, wherein, the observation data defines blocked areas in the environment that restrict vehicle movement within the environment.

21. The autonomous vehicle control system according to claim 16, wherein, the observation data defines one or more vehicle paths, each vehicle path being associated with a path detected by a different vehicle when traveling through the environment of the autonomous vehicle or the path of the autonomous vehicle, and wherein, the one or more processors are configured to: transmit the observation data to the remote real-time map system by transmitting the one or more vehicle paths to the remote real-time map system for controlling the one or more other autonomous vehicles operating in the environment.

22. The autonomous vehicle control system according to claim 16, wherein, the observation data defines sub-optimal actions performed by the autonomous vehicle when traveling through the environment, and wherein, the one or more processors are configured to: transmit the observation data to the remote real-time map system by transmitting the sub-optimal actions to the remote real-time map system for controlling the one or more other autonomous vehicles operating in the environment.

23. A method of operating an autonomous vehicle using an autonomous vehicle control system, comprising: receiving a digital map of a portion of an environment in which the autonomous vehicle operates, the digital map defining a plurality of static elements within the portion of the environment; determining observation data associated with one or more observations collected from the environment based on sensor data collected using one or more sensors of the autonomous vehicle; enhancing the digital map with the observation data to generate an enhanced digital map; using the enhanced digital map to control the autonomous vehicle; and transmitting the observation data to a remote real-time map system for controlling one or more other autonomous vehicles operating in the environment.

24. A method of operating an autonomous vehicle, comprising: receiving, from a remote real-time map system, observation data associated with one or more observations collected from an environment, wherein the observation data defines a plurality of vehicle paths, each vehicle path being associated with a path detected by a different vehicle when traveling through the environment; performing motion planning using the received observation data to generate a motion path for the autonomous vehicle based at least in part on the plurality of vehicle paths; and using the motion path to control the autonomous vehicle.

25. The method according to claim 24, wherein, Associate a first vehicle path among the plurality of vehicle paths with a detected path of a first vehicle detected using one or more sensors of a second vehicle operating within the environment.

26. The method according to claim 25, wherein, The first vehicle is a non-autonomous vehicle.

27. The method according to claim 24, wherein, Associate a plurality of vehicle paths among the plurality of vehicle paths with non-autonomous vehicles.

28. The method according to claim 24, wherein, The observed data further includes sub-optimal actions performed by another autonomous vehicle while traveling through the environment, and wherein, Motion planning using the received observed data to generate a motion path for the autonomous vehicle is at least partially based on the sub-optimal actions.

29. A method, comprising: Conduct a remote assistance session with a first autonomous vehicle operating in an environment, including exchanging situational data and remote assistance operator input between the first autonomous vehicle and a remote assistance system at a remote end; Generate a location-based remote assistance trigger associated with the remote assistance session; And Transmit the location-based remote assistance trigger to a remote real-time map system at the remote end, so that the remote real-time map system forwards the location-based remote assistance trigger to a second autonomous vehicle.

30. The method according to claim 29, wherein, The location-based remote assistance trigger includes a session suggestion, When the second autonomous vehicle meets a location-based criterion associated with the location-based remote assistance trigger, the session suggestion selectively suggests an automatic initiation of a remote assistance session with the second autonomous vehicle.

31. The method according to claim 29, wherein, The location-based remote assistance trigger includes a suggested action to be taken by the second autonomous vehicle when the second autonomous vehicle meets a location-based criterion associated with the location-based remote assistance trigger.

32. The method according to claim 29, further comprising: In the first autonomous vehicle, receive observed data associated with one or more observations collected from the environment from the remote real-time map system, and use the received observed data to control the autonomous vehicle.

33. The method according to claim 29, further comprising: In response to remote assistance operator input, selectively propagate observed data collected in association with the remote assistance system at the remote end to the remote real-time map system.

34. A method, comprising: Receive, from a remote real-time map system, observed data associated with one or more observations collected from an environment in which a first autonomous vehicle operates; Use the observed data to control the first autonomous vehicle; Receive, from the remote real-time map system, a location-based remote assistance trigger generated in association with a remote assistance session between a remote assistance system at a remote end and a second autonomous vehicle; And In response to determining that the first autonomous vehicle meets a location-based criterion associated with the location-based remote assistance trigger, determine the activation of the location-based remote assistance trigger.

35. The method according to claim 34, wherein, the remote assistance session is a first remote assistance session, wherein the location-based remote assistance trigger includes a session suggestion for selectively suggesting an automatic initiation of a second remote assistance session, and the method further includes: when the first autonomous vehicle meets the location-based criteria associated with the location-based remote assistance trigger, automatically initiating the second remote assistance session between the remote assistance system at the remote end and the first autonomous vehicle.

36. The method according to claim 34, wherein, the location-based remote assistance trigger includes a suggested action to be taken by the first autonomous vehicle, and the method further includes: controlling the first autonomous vehicle at least in part based on the suggested action.

37. A real-time map system, comprising: one or more processors; and a memory storing instructions that, when executed by the one or more processors, cause the real-time map system to: maintain observation data associated with a plurality of observations collected by a plurality of autonomous vehicles operating in an environment; maintain a location-based remote assistance trigger generated in association with a remote assistance session conducted between a remote assistance system at a remote end and a first autonomous vehicle among the plurality of autonomous vehicles; and transmit a portion of the observation data and the location-based remote assistance trigger to a second autonomous vehicle among the plurality of autonomous vehicles for controlling the second autonomous vehicle.

Images