Autonomous vehicle system

Abstract

The invention discloses an autonomous vehicle system. An apparatus, the apparatus comprising: at least one interface for receiving sensor data from a plurality of sensors of a vehicle; and one or more processors to: autonomously control driving of the vehicle according to a path plan based on the sensor data; determining that autonomous control of the vehicle should be stopped; sending a handover request to the remote computing system so that the remote computing system remotely controls driving of the vehicle; receiving driving instruction data from the remote computing system; and controlling driving of the vehicle based on the instructions included in the driving instruction data.

Description

This application is a divisional application of the invention patent application entitled "Autonomous Transportation System", with PCT international application number PCT / US2020 / 025474, international application date of March 27, 2020, and application number 202080020333.4 entering the Chinese national phase. Cross-references to related applications

[0001] This application claims the benefit and priority of U.S. Provisional Patent Application No. 62 / 826,955, filed March 29, 2019, entitled “Autonomous Vehicle System”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This disclosure relates generally to the field of computer systems, and more specifically to computing systems for realizing autonomous vehicles. Background Technology

[0003] Some vehicles are configured to operate in autonomous mode, in which they navigate through their environment with little or no input from the driver. Such vehicles typically include one or more sensors configured to sense information about their environment. The vehicle can use this sensed information to navigate around its surroundings. For example, if the sensors detect that the vehicle is approaching an obstacle, it can navigate around the obstacle. Attached Figure Description

[0004] Figure 1 This is a simplified illustration of an example autonomous driving environment according to certain embodiments.

[0005] Figure 2 This is a simplified block diagram illustrating an example implementation of a vehicle (and corresponding onboard computing system) equipped with autonomous driving capabilities according to certain embodiments.

[0006] Figure 3 The figure illustrates an example portion of a neural network conforming to certain embodiments, according to certain embodiments.

[0007] Figure 4 This is a simplified block diagram illustrating example autonomous driving levels that can be supported in various vehicles (e.g., supported by their corresponding onboard computing systems) according to certain embodiments.

[0008] Figure 5 This is a simplified block diagram illustrating an example autonomous driving process that can be implemented in some autonomous driving systems according to certain embodiments.

[0009] Figure 6This is a simplified block diagram illustrating autonomous vehicles and various sensors according to certain embodiments.

[0010] Figure 7 This is a simplified block diagram illustrating communication between systems during the delivery of an example remote valet parking service according to certain embodiments.

[0011] Figure 8 This is a simplified block diagram illustrating information related to collaborative reporting of side-by-side parking incident risks and road condition warnings according to certain embodiments, which can be used to initiate remote valet parking services.

[0012] Figure 9 This is a simplified block diagram illustrating example autonomous vehicle features according to certain embodiments, including vehicle sensors, an AI / machine learning-based autonomous driving stack, and logic supporting the triggering and generation of handover requests to a system capable of providing remote valet parking services.

[0013] Figure 10 The figure illustrates an example safety model driving phase according to certain embodiments.

[0014] Figure 11 This is a diagram of a system for modifying driver input to ensure acceleration in accordance with certain embodiments.

[0015] Figure 12 The training phase of a control-accelerometer according to certain embodiments is described.

[0016] Figure 13 The inference phase of a control-accelerometer according to certain embodiments is described.

[0017] Figure 14 A process for providing acceptable control signals to a vehicle actuation system, according to certain embodiments, is described.

[0018] Figure 15 The training phase for building a context model, according to certain embodiments, is described.

[0019] Figure 16 The training phase for constructing a signal quality metric model, according to certain embodiments, is described.

[0020] Figure 17 The training phase for building a switching-ready model, according to certain embodiments, is described.

[0021] Figure 18 An inference phase for determining a switching decision based on sensor data, according to certain embodiments, is described.

[0022] Figure 19A process for determining whether to transfer control of a vehicle, according to certain embodiments, is described.

[0023] Figure 20 The training phase of a driver state model according to certain embodiments is described.

[0024] Figure 21 The training phase of the handover decision model according to certain embodiments is described.

[0025] Figure 22 Describes the inference phase for determining handover decisions according to certain embodiments.

[0026] Figure 23 A process for generating handover decisions, according to certain embodiments, is described.

[0027] Figure 24 The figure shows a high-level block diagram of a framework for controlling autonomous vehicles according to certain embodiments.

[0028] Figure 25 This is a diagram illustrating an example process for taking over control of an autonomous vehicle according to certain embodiments.

[0029] Figure 26 This is a diagram illustrating an additional example process for taking over control of an autonomous vehicle according to certain embodiments.

[0030] Figure 27 This is an illustration of an example autonomous driving pipeline 2800 for autonomous vehicles to perceive, plan, and perform actions, according to certain embodiments.

[0031] Figure 28 This is a diagram illustrating an example process of a human driver making a takeover request, according to certain embodiments of an autonomous vehicle.

[0032] Figure 29 Various levels of automation and the associated amount of human driver intervention required, according to certain embodiments, are described.

[0033] Figure 30 The figure illustrates a comprehensive cognitive supervision system according to certain embodiments.

[0034] Figure 31 The figure illustrates an example autonomy level transition according to certain embodiments.

[0035] Figure 32 The diagram illustrates an example of the architectural data flow of an autonomous vehicle operating at the L4 level of autonomy, according to certain embodiments.

[0036] Figure 33 The figure illustrates an example of sending a video signal to a driver according to certain embodiments.

[0037] Figure 34 The diagram illustrates a flow of example autonomous vehicle handover according to certain embodiments.

[0038] Figure 35 The diagram illustrates an example of a process for transferring control of an autonomous vehicle to a human driver, according to certain embodiments.

[0039] Figure 36 The figure illustrates an example system 3600 for handing over autonomous vehicles to human drivers according to certain embodiments.

[0040] Figure 37 The diagram illustrates an example route that a vehicle can take from point A to point B according to certain embodiments.

[0041] Figure 38 The diagram illustrates a process that, according to certain embodiments, can be at least partially performed by the handover processing module.

[0042] Figures 39-40 This is a block diagram of an exemplary computer architecture that can be used according to the embodiments disclosed herein. Detailed Implementation

[0043] Figure 1This is a simplified illustration 100 illustrating an example autonomous driving environment. Vehicles (e.g., 105, 110, 115, etc.) may be provided with different levels of autonomous driving capabilities facilitated by onboard computing systems having logic implemented in hardware, firmware, and / or software to achieve corresponding autonomous driving stacks. Such autonomous driving stacks may allow the vehicle to self-control or provide driver assistance to detect roads, navigate from one point to another, detect other vehicles and road actors (e.g., pedestrians (e.g., 135), cyclists, etc.), detect obstacles and hazards (e.g., 120), and road conditions (e.g., traffic, road conditions, weather conditions, etc.), and adjust the control and guidance of the vehicle accordingly. Within this disclosure, a vehicle can be a manned vehicle (e.g., a car, truck, van, bus, motorcycle, train, air transport vehicle, ambulance, etc.) designed to carry one or more human passengers, an unmanned vehicle (e.g., a freight vehicle (e.g., a truck, rail-based vehicle, etc.)) driven with or without human passengers, a vehicle for transporting non-human passengers (e.g., a livestock transport vehicle, etc.), and / or a drone (e.g., a land-based drone or robot or an aerial drone or robot used to move in a driving environment (e.g., to collect information about the driving environment, to assist in the automation of other vehicles, to perform road maintenance tasks, to provide industrial tasks, to provide public safety and emergency response tasks, etc.)). In some implementations, a vehicle can be a system configured to operate alternately in multiple different modes (e.g., a passenger vehicle, an unmanned vehicle, or a drone vehicle), and other examples. A vehicle can "drive" within an environment to move along the ground (e.g., paved or unpaved roads, paths, or landscapes), through water, or through air. In this sense, depending on the implementation, a "highway" or "road" can be embodied as an outdoor or indoor ground-based path, a canal, or a defined aerial boundary. Therefore, it should be understood that the following disclosed and related embodiments can be equally applied to various scenarios and examples of vehicle implementations.

[0044] In some implementations, vehicles within the environment (e.g., 105, 110, 115) can be "connected" by the onboard computing system including a communication module for supporting wireless communication using one or more technologies (e.g., IEEE 802.11 communication (e.g., WiFi), cellular data networks (e.g., 3GPP networks, GSM, CDMA, etc.), 4G, 5G, 6G, Bluetooth, mmWave, ZigBee, Z-wave, etc.), thereby allowing the onboard computing system to connect to and communicate with other computing systems (such as onboard computing systems of other vehicles, roadside units, cloud-based computing systems, or other supporting infrastructure). For example, in some implementations, vehicles (e.g., 105, 110, 115) can communicate with computing systems that provide sensors, data, and services to support the vehicle's own autonomous driving capabilities. For example, as in... Figure 1 As illustrated in the illustrative examples, support is provided for unmanned aerial vehicles 180 (e.g., ground-based and / or aerial), roadside computing devices (e.g., 140), various external (e.g., vehicle-external or “external”) sensor devices (e.g., 160, 165, 170, 175, etc.), and other devices as an autonomous driving infrastructure separate from the computing systems, sensors, and logic implemented on the vehicles (e.g., 105, 110, 115) to support and improve autonomous driving outcomes provided by the vehicles, and other examples. The vehicles can also communicate with other connected vehicles via wireless communication channels to share data and coordinate movement within the autonomous driving environment, and other examples of communication.

[0045] As in Figure 1As illustrated in the examples, autonomous driving infrastructure can comprise a variety of different systems. Such systems can vary depending on location, with more developed roads (e.g., roads officially controlled by a specific municipality or toll plaza, roads in urban areas, road sections known to be problematic for autonomous vehicles, etc.) having a greater number or more advanced supporting infrastructure equipment compared to other road sections. For example, supplementary sensor devices (e.g., 160, 165, 170, 175) may be provided, including sensors for observing road sections and vehicles moving within the environment and generating corresponding data describing or materializing the sensor observations. For example, sensor devices may be embedded within the road itself (e.g., sensor 160), on roadside or elevated signs (e.g., sensor 165 on sign 125), sensors (e.g., 170, 175) attached to electronic roadside equipment or fixed installations (e.g., traffic lights (e.g., 130), electronic road signs, electronic bulletin boards, etc.), dedicated roadside units (e.g., 140), and other examples. Sensor devices may also include communication capabilities to transmit the sensor data they collect directly to nearby connected vehicles or to fog-based or cloud-based computing systems (e.g., 140, 150). Vehicles may acquire sensor data collected by external sensor devices (e.g., 160, 165, 170, 175, 180) or make observations or recommendations specific to data generated by other systems (e.g., 140, 150) based on sensor data from these sensor devices (e.g., 160, 165, 170, 175, 180), and use this data in sensor fusion, inference, path planning, and other tasks performed by onboard autonomous driving systems. In some cases, such external sensors and sensor data may actually reside within the vehicle, such as in the form of aftermarket sensors attached to the vehicle, or personal computing devices carried or worn by vehicle occupants (e.g., smartphones, wearable devices, etc.). Other road actors, including pedestrians, cyclists, drones, unmanned aerial vehicles, robots, and e-motorcycles, may also be provided with or carry sensors to generate sensor data describing the autonomous driving environment. This sensor data may be used and consumed by autonomous vehicles, cloud-based or fog-based support systems (e.g., 140, 150), other sensor devices (e.g., 160, 165, 170, 175, 180), and other examples.

[0046] Because autonomous transportation systems can possess varying levels of functionality and complexity, the supporting infrastructure may need to complement not only the sensing capabilities of some vehicles but also computing and machine learning capabilities to enable autonomous driving. For example, computational resources and autonomous driving logic used to facilitate the training and use of machine learning models may be housed on an onboard computing system, either wholly or partially on both the onboard system and external systems (e.g., 140, 150). For instance, connected vehicles may communicate with roadside units, edge systems, or cloud-based devices (e.g., 140) located locally on specific segments of the road. These devices (e.g., 140) may be able to provide data (e.g., sensor data aggregated from local sensors (e.g., 160, 165, 170, 175, 180) or data reported from sensors of other vehicles), perform computations (as a service) on the data provided by the vehicles to (e.g., based on sensor data collected at device 140 or from nearby sensor devices) supplement the vehicle's own capabilities and / or push information to passing or approaching vehicles. Connected vehicles (e.g., 105, 110, 115) may also, or alternatively, communicate with a cloud-based computing system (e.g., 150) that can provide similar memory, sensing, and computing resources to enhance those available at the vehicle. For example, the cloud-based system (e.g., 150) may collect sensor data from multiple devices at one or more locations and use that data to build and / or train a machine learning model that can be used at the cloud-based system (to provide the results to the individual vehicles (e.g., 105, 110, 115) communicating with the cloud-based system 150 or to push the results to the vehicles for use by their onboard systems), and other example implementations. Access points (e.g., 145) may be positioned within the environment to facilitate communication between cloud-based systems (e.g., 150) and various vehicles (e.g., 105, 110, 115) via one or more local area networks (LANs) or wide area networks (WANs) (e.g., 155). These access points include, for example, cellular phone towers, roadside units, network access points installed to various roadside infrastructures, access points provided by adjacent vehicles or buildings, and other access points. Through such infrastructure and computing systems, it should be understood that the examples, features, and solutions discussed herein can be implemented holistically by one or more of such in-vehicle computing systems, fog-based computing devices or edge computing devices, or cloud-based computing devices, or holistically by a combination of the foregoing devices through communication and collaboration between these systems.

[0047] Generally, the terms "server," "client," "computing device," "network element," "host," "platform," "sensor device," "edge device," "autonomous driving system," "autonomous vehicle," "fog-based system," "cloud-based system," and "system" discussed herein can generally include electronic computing devices operable to receive, transmit, process, store, and manage data and information associated with an autonomous driving environment. As used in this document, the terms "computer," "processor," "processor device," or "processing device" are intended to cover any suitable processing apparatus, including central processing units (CPUs), graphics processing units (GPUs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), tensor processors, and other matrix arithmetic processors, among others. For example, elements of a single device shown as an environment may be implemented using multiple computing devices and processors, such as a server pool comprising multiple server computers. Furthermore, any, all, or some of the computing devices may be adapted to run any operating system (including Linux, UNIX, Microsoft Windows, Apple OS, Apple iOS, Google Android, Windows Server, etc.) and to run virtual machines that virtualize the execution of a particular operating system (including custom and proprietary operating systems).

[0048] The functionality of any of the processes, methods, procedures (or parts thereof) described below or illustrated in the accompanying drawings, or any of the various components described herein or illustrated in the accompanying drawings, can be performed by any suitable computational logic, such as one or more modules, engines, blocks, units, models, systems, or other suitable computational logic. The terms "module," "engine," "block," "unit," "model," "system," or "logic" herein can refer to hardware, firmware, software, and / or combinations of each for performing one or more functions. As an example, a module, engine, block, unit, model, system, or logic may include hardware such as a microcontroller or processor associated with a non-transitory medium for storing code suitable for execution by the microcontroller or processor. Therefore, in one embodiment, a reference to a module, engine, block, unit, model, system, or logic may refer to hardware specifically configured to identify and / or execute code to be stored on a non-transitory medium. Furthermore, in another embodiment, the use of "module," engine, block, unit, model, system, or logic" refers to a non-transitory medium comprising code specifically suitable for execution by a microcontroller or processor to perform a predetermined operation. And, as can be inferred, in yet another embodiment, the terms module, engine, block, unit, model, system, or logic may refer to a combination of hardware and non-transitory media. In various embodiments, a module, engine, block, unit, model, system, or logic may include a microprocessor or other processing element operable to execute software instructions, discrete logic such as an application-specific integrated circuit (ASIC), a programmable logic device such as a field-programmable gate array (FPGA), a memory device containing instructions, a combination of logic devices (e.g., as would be found on a printed circuit board), or other suitable hardware and / or software. A module, engine, block, unit, model, system, or logic may include one or more gate circuits or other circuit components, which may be implemented by, for example, transistors. In some embodiments, a module, engine, block, unit, model, system, or logic may be entirely embodied as software. Software may be embodied as software packages, code, instructions, instruction sets, and / or data recorded on a non-transitory computer-readable storage medium. Firmware may be embodied as hard-coded (e.g., non-volatile) code, instructions or instruction sets, and / or data in a memory device. Furthermore, the logical boundaries illustrated as separate are often distinct and potentially overlapping. For example, the first and second modules (or multiple engines, blocks, units, models, systems, or logic) may share hardware, software, firmware, or a combination thereof, while potentially retaining some independent hardware, software, or firmware.

[0049] The processes, methods, and procedures described below and illustrated in the accompanying drawings are merely illustrative of functions that may be performed in a particular embodiment. In other embodiments, additional functions may be performed in the processes, methods, and procedures. The various embodiments of this disclosure are contemplated for any suitable signaling mechanism to implement the functions described herein. Some of the functions illustrated herein may be repeated, combined, modified, or deleted within the processes, methods, and procedures as appropriate. Furthermore, functions may be performed in any suitable order within the processes, methods, and procedures without departing from the scope of the particular embodiments.

[0050] Now for reference Figure 2 A simplified block diagram 200 illustrates an example implementation of a vehicle (and corresponding onboard computing system) 105 equipped with autonomous driving capabilities. In one example, the vehicle 105 may be equipped with one or more processors 202, such as a central processing unit (CPU), graphics processing unit (GPU), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), digital signal processor (DSP), tensor processor, and other matrix arithmetic processors, among others. Such processors 202 may be coupled to an integrated hardware accelerator device (e.g., 204) or have an integrated hardware accelerator device that may be provided with hardware for accelerating certain processing and memory access functions, such as processing of machine learning inference or training (including any of the machine learning inference or training described below), processing of specific sensor data (e.g., camera image data, LiDAR point clouds, etc.), performing functions related to certain arithmetic functions involving autonomous driving (e.g., matrix arithmetic, convolution arithmetic, etc.), among others. One or more memory elements (e.g., 206) may be provided to store machine-executable instructions, as well as machine learning models (e.g., 256), sensor data (e.g., 258), and other data received, generated, or used in relation to autonomous driving functions to be performed by the vehicle. These machine-executable instructions implement all or part of any of the modules or sub-modules of the autonomous driving stack implemented on the vehicle. Various communication modules (e.g., 212) may also be provided and implemented in hardware circuitry and / or software to enable communication capabilities used by the vehicle system to communicate with other external computing systems via one or more network channels employing one or more network communication technologies. These various processors 202, accelerators 204, memory devices 206, and network communication modules 212 may be interconnected on the vehicle system via one or more interconnect structures or links (e.g., 208), such interconnect structures or links utilizing peripheral component interconnect fast (PCIe), Ethernet, OpenCAPI, etc. TM Gen-Z TMUPI, Universal Serial Bus (USB), Cache Coherence Interconnect for Accelerators (CCIX) TM ), Advanced Micro Semiconductor TM (AMD) TM Infinity TM Common Communication Interface (CCI) or Qualcomm TM Centriq TM Interconnection structures or links, etc.

[0051] continue Figure 2 For example, an example vehicle (and a corresponding onboard computing system) 105 may include onboard processing system 210, driving control devices (e.g., 220), sensors (e.g., 225), and (multiple) user / occupant interfaces (e.g., 230), etc., in hardware and / or software implementing the functions of an autonomous vehicle. For example, in some implementations, onboard processing system 210 may implement autonomous driving stacking and process flow (e.g., as... Figure 5 All or part of the examples shown and discussed herein. The autonomous driving stack can be implemented in hardware, firmware, or software. A machine learning engine 232 may be provided to combine one or more autonomous functions and features (such as those discussed in the examples herein) provided and implemented at or for the vehicle to utilize various machine learning models (e.g., 256) provided at the vehicle 105. Such machine learning models 256 may include artificial neural network models, convolutional neural networks, decision tree-based models, support vector machines (SVMs), Bayesian models, deep learning models, and other example models. In some implementations, the example machine learning engine 232 may include one or more model trainer engines 252 to participate in the training (e.g., initial training, continued training, etc.) of one or more machine learning models 256. One or more inference engines 254 may also be provided to derive various inferences, predictions, classifications, and other results using the trained machine learning models 256. In some embodiments, the machine learning model training or inference described herein may be performed outside the vehicle (e.g., by computing system 140 or 150).

[0052] Multiple learning engines 232 located at the vehicle can be used to support the onboard processing system 210 and provide results used by other logical components and modules of the onboard processing system 210, thereby enabling autonomous driving stacking and other features related to autonomous driving. For example, the data collection module 234 may be provided with logic for determining from which source data to collect (e.g., inputs for training or use of various machine learning models 256 used by the vehicle). For example, specific sources (e.g., internal sensors (e.g., 225) or external sensors (e.g., 115, 140, 150, 180, 215, etc.) may be selected, and the frequency and fidelity at which data can be sampled may be chosen. In some cases, such selection and configuration may be made at least partially autonomously by the data collection module 234 using one or more corresponding machine learning models (e.g., to collect data as appropriate, taking into account specific detected scenarios).

[0053] Sensor fusion module 236 can also be used to manage the use and processing of various sensor inputs utilized by the machine learning engine 232 and other modules (e.g., 238, 240, 242, 244, 246, etc.) of the in-vehicle processing system. One or more sensor fusion modules (e.g., 236) may be provided, which can derive outputs from multiple sensor data sources (e.g., on or outside the vehicle). Sources can be homogeneous or heterogeneous (e.g., multiple inputs from multiple instances of sensors of a common type, or multiple inputs from instances of multiple different types of sensors). Example sensor fusion module 236 may apply direct fusion, indirect fusion, and other example sensor fusion techniques. In some cases, in conjunction with providing autonomous driving capabilities or other functions, the output of sensor fusion may be fed as input (along with potentially additional inputs) to another module and / or one or more machine learning models of the in-vehicle processing system, as described in the example solutions discussed herein.

[0054] In some examples, a perception engine 238 may be provided, which can take various sensor data (e.g., 258) as input to perform object recognition and tracking of detected objects, etc., corresponding to autonomous perception of the environment encountered (or to be encountered) by the vehicle 105. In some instances, the various sensor data includes data from external sources and / or sensor fusion module 236. The perception engine 238 may use deep learning, such as through one or more convolutional neural networks and other machine learning models 256, to perform object recognition from the sensor data input. Object tracking may also be performed to autonomously estimate whether an object is moving from the sensor data input, and if so, to estimate the trajectory along which the object is moving. For example, after a given object is identified, the perception engine 238 may detect how the given object moves relative to the vehicle. Such functionality may be used, for example, to detect objects moving within the environment that may affect the path of the vehicle on a road, such as other vehicles, pedestrians, wild animals, cyclists, etc., and other examples.

[0055] In one implementation, the positioning engine 240 may also be included within the onboard processing system 210. In some cases, the positioning engine 240 may be implemented as a subcomponent of the perception engine 238. The positioning engine 240 may also utilize one or more machine learning models 256 and sensor fusion (e.g., sensor fusion of LiDAR and GPS data) to determine the high-confidence location of the vehicle and the space it occupies within a given physical space (or “environment”).

[0056] The vehicle 105 may further include a path planner 242, which may utilize the results of various other modules such as data collection (e.g., 234), sensor fusion (e.g., 236), perception engines (e.g., 238), and localization engines (e.g., 240), etc. (e.g., recommendation engine 244), to determine path planning and / or motion planning for the vehicle. This path planning and / or motion planning may be used by a driving control device (e.g., 220) to control the driving of the vehicle 105 within an environment. For example, the path planner 242 may utilize these inputs and one or more machine learning models to determine the probabilities of various events within the driving environment to determine an effective real-time plan to be taken within that environment.

[0057] In some implementations, vehicle 105 may include one or more recommendation engines 244 for generating various recommendations from sensor data generated by sensors (e.g., 225) of vehicle 105 itself and sensor data from external sensors (e.g., on sensor devices 115, 180, 215, etc.). Some recommendations may be determined by recommendation engine 244, and these recommendations may be provided as input to other components of the autonomous driving stack of the vehicle to influence the determinations made by these components. For example, a recommendation may be determined such that, when considered by path planner 242, it causes path planner 242 to deviate from its normally determined decision or plan (if not a recommendation). Recommendations may also be generated by recommendation engine (e.g., 244) based on considerations of occupant comfort and experience. In some cases, internal features within the vehicle may be predictively and autonomously manipulated based on these recommendations (which are determined from sensor data (e.g., 258) captured by sensors of the vehicle and / or external sensors).

[0058] As described above, some vehicle implementations may include a user / occupant experience engine (e.g., 246) that uses sensor data and outputs from other modules within the vehicle's autonomous driving stack, based on observations captured by sensor data (e.g., 258), to control the vehicle's control unit to alter driving maneuvers and influence changes to the vehicle's cabin environment to improve the occupant experience. In some instances, aspects of the user interface (e.g., 230) on the vehicle that enable users to interact with the vehicle and its autonomous driving environment may be enhanced. In some cases, informative presentations may be generated and provided via user displays (e.g., audio, visual, and / or haptic presentations) to help influence and improve the occupant experience within the vehicle (e.g., 105), and other examples use this approach.

[0059] In some cases, a system manager 250 may also be provided, which monitors information collected by various sensors on the vehicle to detect problems related to the performance of the vehicle's autonomous driving system. For example, the system manager 250 may detect computational errors, sensor downtime and problems, availability and quality of communication channels (e.g., provided via communication module 212), vehicle system checks (e.g., problems related to motors, transmissions, batteries, cooling systems, electrical systems, tires, etc.), or other operational events. Such problems may be identified in system report data generated by the system manager 250, and in some cases, these problems may be used as input to machine learning model 256 and relevant autonomous driving modules (e.g., 232, 234, 236, 238, 240, 242, 244, 246, etc.) so that vehicle system health and problems can be considered in the autonomous driving functions of vehicle 105 along with other information collected in sensor data 258.

[0060] In some implementations, the autonomous driving stack of vehicle 105 may be coupled to driving control unit 220 to influence how the vehicle is driven. Driving control unit 620 includes steering control (e.g., 260), accelerator / throttle control (e.g., 262), braking control (e.g., 264), signal transmission control (e.g., 266), and other examples. In some cases, the vehicle may also be controlled entirely or partially based on user input. For example, a user interface (e.g., 230) may include driving control units (e.g., a physical or virtual steering wheel, accelerator, brakes, clutch, etc.) to allow a human driver to take control from the autonomous driving system (e.g., by handing over or after driver assistance actions). Other sensors may be used to accept user / occupant input, such as voice detection 292, posture detection camera 294, and other examples. The user interface (e.g., 230) may capture occupant-user expectations and intentions, and the autonomous driving stack of vehicle 105 may use these as additional input when controlling the driving of the vehicle (e.g., driving control unit 220). In some implementations, driving control may be managed by an external computing system, such as in cases where occupants use external devices (e.g., smartphones or tablets) to provide driving direction or control, or in remote valet parking services where an external driver or system takes over control of the vehicle (e.g., based on an emergency), and other example implementations.

[0061] As discussed above, the autonomous driving stack of a vehicle can utilize various sensor data (e.g., 258) generated by various sensors mounted on and outside the vehicle. As an example, vehicle 105 may have an array of sensors 225 to collect various information relating to the external and surrounding environment of the vehicle, the vehicle system state, the internal conditions of the vehicle, and other information usable by modules of the vehicle's processing system 210. For example, such sensors 225 may include a Global Positioning (GPS) sensor 268, a Light Detection and Ranging (LIDAR) sensor 270, a two-dimensional (2D) camera 272, a three-dimensional (3D) or depth-sensing camera 274, an acoustic sensor 276, an Inertial Measurement Unit (IMU) sensor 278, a thermal sensor 280, an ultrasonic sensor 282, a biosensor 284 (e.g., facial recognition, voice recognition, heart rate sensor, body temperature sensor, emotion detection sensor, etc.), a radar sensor 286, a weather sensor (not shown), and other example sensors. Such sensors can be used in combination to determine various properties and conditions of the environment in which the vehicle operates (e.g., weather, obstacles, traffic, road conditions, etc.), passengers within the vehicle (e.g., passenger or driver perception or alertness, passenger comfort or mood, passenger health or physiological condition, etc.), other contents of the vehicle (e.g., packages, livestock, cargo, luggage, etc.), subsystems of the vehicle, and other examples. Sensor data 258 may also (or alternatively) be provided by sensors not integrally coupled to the vehicle, including sensors on other vehicles (e.g., 115) that communicate with vehicle 105 via vehicle-to-vehicle communication or other technologies, sensors on ground-based or airborne drones 180, sensors on user devices 215 (e.g., smartphones or wearable devices) carried by human users inside or outside vehicle 105, and sensors that are mounted or provided with other roadside elements (such as roadside units (e.g., 140), road signs, traffic lights, streetlights, etc.). Sensor data from such external sensor devices can be provided directly from the sensor devices to the vehicle, or through data aggregation devices, or as a result generated by other computing systems (e.g., 140, 150) based on these sensors, and so on.

[0062] In some implementations, the autonomous vehicle system 105 may interface with other computing systems and utilize information and services provided by those other computing systems to enhance, implement, or otherwise support the autonomous driving capabilities of the vehicle 105. In some instances, some autonomous driving features (including some of the example solutions discussed herein) may be implemented through services, computational logic, machine learning models, data, or other resources of computing systems external to the vehicle. When such external systems are unavailable to the vehicle, these features may be disabled, at least temporarily. For example, external computing systems may be available and utilized, which may be hosted in roadside units or fog-based edge devices (e.g., 140), other (e.g., higher-level) vehicles (e.g., 115), and cloud-based systems 150 (e.g., accessible via various network access points (e.g., 145)). The roadside unit 140 or cloud-based system 150 (or other collaborative systems with which the vehicle (e.g., 105) interacts) may include all or part of the logic illustrated as belonging to the example onboard processing system (e.g., 210), and potentially additional functionality and logic. For example, cloud-based computing systems, roadside units 140, or other computing systems may include machine learning engines that support either or both of the model training and inference engine logic. For instance, such external systems may possess more advanced computing resources and more sophisticated or recent machine learning models, allowing these services to deliver results superior to those natively generated on the vehicle's processing system 210. For example, the onboard processing system 210 may rely on machine learning training, machine learning inference, and / or machine learning models provided via cloud-based services for certain tasks and scenarios. Indeed, it will be appreciated that in some implementations, one or more of the modules discussed and illustrated as belonging to vehicle 105 may be alternatively or redundantly housed in cloud-based computing systems, fog-based computing systems, or other computing systems supporting autonomous driving environments.

[0063] The embodiments described herein may utilize one or more machine learning models to perform autonomous vehicle stacking functionality (or other functionality described herein). The machine learning models may be executed by a computing system to progressively improve the performance of a particular task. In some embodiments, the parameters of the machine learning model may be tuned based on training data during a training phase. The trained machine learning model may then be used to make predictions or decisions based on the input data during an inference phase.

[0064] The machine learning models described in this paper can take any suitable form or utilize any suitable techniques. For example, any of these machine learning models can utilize supervised learning, semi-supervised learning, unsupervised learning, or reinforcement learning techniques.

[0065] In supervised learning, a model is built using a training dataset that contains both inputs and corresponding expected outputs. Each training instance may include one or more inputs and the expected output. Training may involve iterating through training instances and using an objective function to teach the model to predict the output for new inputs. In semi-supervised learning, the expected output may be missing from the input portion of the training set.

[0066] In unsupervised learning, models can be built from datasets that contain only inputs and no desired outputs. Unsupervised models can be used to find structures in data (e.g., grouping or clustering of data points) by discovering patterns within the data. Techniques that can be implemented in unsupervised learning models include, for example, self-organizing graphs, nearest neighbor mappings, k-means clustering, and singular value decomposition.

[0067] Reinforcement learning models can be given positive or negative feedback to improve accuracy. They can attempt to maximize one or more objectives / rewards. Techniques that can be implemented in reinforcement learning models include, for example, Q-learning, temporal difference (TD), and deep adversarial networks.

[0068] The embodiments described herein may utilize one or more classification models. In a classification model, the output may be limited to a finite set of values. A classification model may output a class for an input set having one or more input values. References to classification models herein may be used to conceive of models implementing any one or more of the following techniques: linear classifiers (e.g., logistic regression or Naive Bayes classifiers), support vector machines, decision trees, boosting trees, random forests, neural networks, or nearest neighbors.

[0069] The embodiments described herein may utilize one or more regression models. A regression model may output numerical values ​​from a continuous range based on an input set having one or more values. References to regression models herein are conceivable to implement models that employ any one or more of the following techniques (or other suitable techniques): linear regression, decision trees, random forests, or neural networks.

[0070] In various embodiments, any of the machine learning models discussed herein may utilize one or more neural networks. A neural network may comprise a loosely modeled set of neural units after constructing a biological brain, which comprises a large cluster of neurons connected by synapses. In a neural network, neural units are connected to other neural units via links, which may have an excitatory or inhibitory effect on the activation state of the connected neural units. A neural unit may use the values ​​of its inputs to perform functions to update the membrane potential of the neural unit. When a threshold associated with the neural unit is exceeded, the neural unit may propagate a spike signal to the connected neural units. Neural networks can be trained or otherwise tuned to perform a variety of data processing tasks (including tasks performed by stacked autonomous vehicles), such as computer vision tasks, speech recognition tasks, or other suitable computational tasks.

[0071] Figure 3 The diagram illustrates an example portion of a neural network 300 according to certain embodiments. The neural network 300 includes neural units X1-X9. Neural units X1-X4 are input neural units that respectively receive main inputs I1-I4 (which can remain constant as the neural network 300 processes the output). Any suitable main input can be used. As an example, when the neural network 300 performs image processing, the main input value can be the value of a pixel from an image (and the main input value can remain constant while processing the image). As another example, when the neural network 300 performs speech processing, the main input value applied to a particular input neural unit can change over time based on variations in the input speech.

[0072] Although Figure 3 Specific topologies and connectivity schemes are illustrated, but the teachings of this disclosure can be used in neural networks with any suitable topology and / or connectivity. For example, the neural network can be a feedforward neural network, a recurrent network, or any other neural network with any suitable connectivity between neurons. As another example, although the neural network is depicted as having input layers, hidden layers, and output layers, the neural network can have any suitable layers arranged in any suitable manner. In the depicted embodiments, the link between each pair of neurons has a synaptic weight, which indicates the strength of the relationship between the two neurons. The synaptic weight is depicted as WXY, where X indicates a presynaptic neuron and Y indicates a postsynaptic neuron. The link between neurons may have an excitatory or inhibitory effect on the activation state of the connected neurons. For example, depending on the value of W15, a spike propagating from X1 to X5 can increase or decrease the membrane potential of X5. In various embodiments, the connection can be directional or non-directional.

[0073] In various embodiments, during each time step of the neural network, a neural unit can receive any suitable input, such as a bias value or one or more input spikes from one or more neural units connected to it via corresponding synapses (this set of neural units is referred to as the fan-in neural unit of that neural unit). The bias value applied to a neural unit can be a function of the main input applied to the input neural unit and / or some other value applied to the neural unit (e.g., a constant value that can be adjusted during the training or other operation of the neural network). In various embodiments, each neural unit can be associated with its own bias value, or bias values ​​can be applied to multiple neural units.

[0074] A neural unit can perform functions using the values ​​of its inputs and its current membrane potential. For example, the input can be added to the neural unit's current membrane potential to generate an updated membrane potential. As another example, a nonlinear function such as the sigmoid transfer function can be applied to the input and the current membrane potential. Any other suitable function can be used. The neural unit then updates its membrane potential based on the function's output.

[0075] Turn Figure 4The diagram shows a simplified block diagram 400 illustrating example levels of autonomous driving that can be supported in various vehicles (e.g., by their corresponding onboard computing systems). For example, a range of levels can be defined (e.g., L0-L5 (405-435)), where Level 5 (L5) corresponds to a vehicle with the highest level of autonomous driving capabilities (e.g., fully automated) and Level 0 (L0) corresponds to the lowest level of autonomous driving capabilities (e.g., no automation). For example, an L5 vehicle (e.g., 435) may have a fully autonomous computing system capable of providing autonomous driving performance equal to or better than that provided by a human driver in every driving scenario. An L4 vehicle (e.g., 430) can also be considered fully autonomous and capable of autonomously performing safety-critical driving functions and effectively monitoring road conditions throughout the journey from the starting point to the destination. L4 can differ from L5 in that the autonomy of L4 is confined to the vehicle's "operational design domain," which may not include all driving scenarios. Level 3 vehicles (e.g., 420) provide autonomous driving capabilities to fully transfer safety-critical functions to the vehicle in a specific set of traffic and environmental conditions, but still require human driver involvement and availability to handle driving in all other scenarios. Accordingly, Level 3 vehicles can provide handover protocols to orchestrate the transfer of control from the human driver to the autonomous driving stack and the transfer of control back to the human driver from the autonomous driving stack. Level 2 vehicles (e.g., 415) provide driver assistance features that allow the driver to occasionally disengage from physical operation of the vehicle, so that both the driver's hands and feet can be periodically disengaged from physical control of the vehicle. Level 1 vehicles (e.g., 410) provide driver assistance for one or more specific functions (e.g., steering, braking, etc.), but still require constant driver control over most functions of the vehicle. Level 0 vehicles can be considered non-autonomous—the human driver controls all driving functions of the vehicle (although such vehicles can still passively participate in autonomous driving environments, such as by providing sensor data to higher-level vehicles, using sensor data to enhance GPS and infotainment services within the vehicle, etc.). In some implementations, a single vehicle may support operation at multiple levels of autonomy. For example, a driver may control and select which supported level of autonomy (e.g., L4 or lower) is used during a given trip. In other cases, the vehicle may autonomously switch between levels, for example, based on the condition of the autonomous driving system affecting the road or the vehicle. For instance, in response to the detection that one or more sensors have been damaged, an L5 or L4 vehicle may switch to a lower mode (e.g., L2 or lower) to allow human occupants to participate, given the sensor problem, and other examples.

[0076] Figure 5 This is a simplified block diagram 500 illustrating an example autonomous driving process that can be implemented in some autonomous driving systems. For example, an autonomous driving process implemented in an autonomous (or semi-autonomous) vehicle may include a sensing and perception phase 505, a planning and decision-making phase 510, and a control and action phase 515. During the sensing and perception phase 505, data is generated by various sensors and collected for use by the autonomous driving system. In some instances, data collection may include data filtering and receiving sensors from external sources. This phase may also include sensor fusion operations, object recognition, and other perception tasks, such as localization performed using one or more machine learning models. The planning and decision-making phase 510 may utilize sensor data and the results of various perception operations to make probabilistic predictions of the road ahead (multiple roads) and determine real-time path planning based on these predictions. The planning and decision-making phase 510 may additionally include making decisions about path planning as a response to detected obstacles and other events to determine whether and what actions to take to safely navigate the determined path. Based on the path planning and decision-making in the planning and decision-making stage 510, the control and action stage 515 can translate these decisions into actions through actuators to manipulate the driving control devices and auxiliary control devices, including steering devices, acceleration devices, braking devices, and auxiliary control devices such as turn signals, sensor cleaners, windshield wipers, headlights, etc.

[0077] While all the functions required for autonomous vehicle operation can be provided natively by the onboard computing system (and updated as needed via periodic communication through wired or wireless home-based or garage-based network connections), the advancement of wireless communication technologies and speeds may lead to some autonomous vehicle implementations relying more heavily on communication from external data and computing resources (e.g., outside the vehicle or not natively integrated with it). For example, with the advent of 5G operator networks and the expansion of 4G LTE coverage, connected autonomous vehicles and vehicle-to-the-world (V2X) systems are becoming more readily available. For instance, safety, provided natively by the autonomous driving system on the vehicle, can be supplemented, enhanced, and augmented by systems outside the vehicle to provide both enhanced and crowdsourced intelligence, as well as redundancy (such as through real-time high-reliability applications).

[0078] Autonomous vehicles can communicate with and be guided by external computing systems. Such control can include low-level controls, such as push over-the-air (OTA) updates, where the vehicle can receive software and / or firmware updates from a remote control / maintenance center (e.g., belonging to the original equipment manufacturer (OEM) or supplier of the vehicle or autonomous driving system), rather than bringing the vehicle to a maintenance center for manual intervention by a technician. In other applications with higher levels of control, complete control of the autonomous vehicle can be transferred to a remote user / virtual driver on an external computing system or remote computing terminal. For example, remote control can provide services such as on-demand "remote valet parking," for instance, when it is impractical or undesirable for the autonomous vehicle to transfer control to passengers; assisting a vehicle whose autonomous driving system is struggling to navigate a specific section of a route accurately, efficiently, or safely; or assisting in handling pull-over events or other unmanageable autonomous vehicle situations.

[0079] In some implementations, when an autonomous vehicle encounters a situation or event that it does not know how to handle reliably and safely, the vehicle can be programmed to initiate a pullover event, where the autonomous driving system guides the vehicle off the road (e.g., to the shoulder, to a parking space, etc.). In the future, as more autonomous vehicles are found on the road, an event that causes one autonomous vehicle to initiate a pullover could similarly affect other adjacent autonomous vehicles, potentially leading to multiple pullovers causing additional congestion and road blockages, and potentially paralyzing the roads and the autonomous vehicles on those roads. While some instances may allow handover events from the autonomous driving system to human passengers to deal with situations that lead to pullovers, other implementations may trigger remote valet parking services (e.g., when the vehicle is without passengers (e.g., drone vehicles, vehicles in progress that have triggered remote valet parking services to their passengers using remote summoning functionality, etc.)), as well as other example situations and implementations.

[0080] As described above, some implementations of autonomous vehicles can support remote valet parking, allowing the transfer of driving control (from the vehicle's autonomous driving system) to a remote computing system controlled via a network. For example, when an autonomous vehicle faces a situation it cannot handle (e.g., sensors malfunction, new road conditions are unknown to the vehicle, onboard systems cannot make decisions, etc.), remote control can be triggered on demand by the autonomous vehicle. Such remote control can also be provided to the vehicle in emergency situations where it requests it. Remote valet parking services may involve a human remotely seated in a control and maintenance center equipped with user terminal systems for remotely controlling the vehicle. Such systems can be used to mitigate edge cases where, given a lack of actionable information about itself or its environment, an autonomous vehicle might be unable to maneuver and may pull over or remain stationary. Remote valet parking systems can also be equipped with the ability to receive information from autonomous systems (e.g., the autonomous system would provide a view of the road the vehicle is navigating, information about the vehicle's system status, passenger status, etc.), but can still operate independently of the vehicle's autonomous driving system. This independence allows the remote valet parking service itself to function even under conditions of complete or extensive sensor failure in the autonomous vehicle, as well as other example use cases, benefits, and implementations.

[0081] For example, such as Figure 6As shown in the simplified block diagram 600, the autonomous vehicle 105 may include various sensors (e.g., 620, 625, 630, etc.) and autonomous driving logic for enabling the autonomous vehicle to drive automatically in various environments. As described above, in some cases, the autonomous vehicle (or at the request of passengers within the autonomous vehicle) may determine portions of the route in the route planning that the autonomous driving system of vehicle 105 cannot reliably, ideally, or safely navigate. The autonomous vehicle 105 may include communication capabilities for interfacing with one or more networks (e.g., 155) and enabling data exchange between vehicle 105 and one or more computing systems implementing the remote valet parking service 605. The remote valet parking service 605 can provide multiple user terminal devices that allow a virtual driver user to observe the conditions around vehicle 105 based on sensor data (e.g., camera views or other sensor information) provided by sensors on the vehicle (e.g., 620, 625, 630, etc.) or sensors on other devices (e.g., roadside systems (e.g., 130), aerial or ground-based drones (e.g., 180), and even sensors from other adjacent vehicles) (e.g., 175). The virtual driver can then provide input at the remote valet parking terminal to deliver corresponding low-latency, high-priority data (via network 155) to vehicle 105 to control the steering, acceleration, and braking of vehicle 105.

[0082] In some situations, vehicle 105 may automatically request intervention and transfer control to remote valet parking service 605. In some cases, the request may be reactive (e.g., in response to a pullover event, sensor malfunction, or emergency), while in others, the request may be sent to preemptively cause remote valet parking service 605 to take over control of the vehicle (based on predictions of a potential pullover event or other difficulties considering conditions ahead on the route). Vehicle 105 may utilize sensor data from its own sensors (e.g., 620, 625, 630, etc.), as well as data from other sensors and devices (e.g., 130, 180, etc.), and backend autonomous driving support services (e.g., cloud-based service 150), to use one or more machine learning models to determine the circumstances under which control should be transferred to remote valet parking service 605.

[0083] In some cases, multiple remote valet parking services may exist, which can be utilized by any of several different autonomous vehicles. In fact, multiple autonomous vehicles can simultaneously connect to and be controlled by a single remote valet parking service (e.g., guided by different remote drivers for each vehicle). In some cases, one remote valet parking service may announce more availability than another. In some cases, remote valet parking service quality ratings can be maintained. In still other cases, connection quality and speed information can be maintained to identify the real-time connection status of each of the multiple different remote valet parking services. Therefore, in addition to detecting the need for or potential remote handover, the autonomous vehicle (e.g., 105) can also consider such inputs to determine which of the many potentially available alternative remote valet parking services can be used and requested. In some implementations, such as instances where a vehicle is associated with a specific remote valet parking service (e.g., through proactive subscriptions to remote valet parking services from a specific provider, remote valet parking services associated with the vehicle manufacturer or its autonomous driving system, and other considerations), the choice will be direct.

[0084] Furthermore, remote valet parking services can also tailor services to individual autonomous vehicles (e.g., 105) and their owners and passengers based on various attributes detected by the remote valet parking service (e.g., information included in the handover request, information collected from sensor data received in conjunction with handover or remote control, etc.). For example, a customized driving assistance user interface and controls can be provided and presented to the virtual driver of the remote valet parking service based on the brand and model of the controlled vehicle, the version and implementation of the vehicle's autonomous driving system, which sensors on the vehicle remain operational and reliable, the specific circumstances facilitating the handover (e.g., requiring a professional remote driver to assist in fault detection and navigating the vehicle out of difficult extreme situations), and other examples.

[0085] In some implementations, remote valet parking services can be provided by government agencies as a public service. In other implementations, remote valet parking services can be provided by privately-owned commercial enterprises. Therefore, in conjunction with the remote valet parking service provided in conjunction with a trip using a given vehicle (e.g., 105), metrics can be automatically collected and corresponding data (e.g., through sensors or monitors on either or both of the vehicle (e.g., 105) and the remote valet parking system 605) can be generated to describe the remote valet parking service provided. Such metrics and data can describe characteristics of remote valet parking services such as the severity of the condition triggering the service (e.g., more difficult problems require higher fees), mileage traveled under the control of the remote valet parking service, time spent under its control, the specific virtual driver and vehicle used to facilitate the service, the source and quantity of external data used by the service (e.g., the amount of data requested and collected from sources other than sensors (e.g., 175, 180),) and other metrics that can be considered and used to determine the fees charged by the remote virtual service for its services. In some cases, the fees may be paid by the vehicle owner, vehicle manufacturer, vehicle warranty provider, supplier of the vehicle's autonomous driving system, etc., or shared among them. In some cases, liability for remote valet parking fees can be automatically determined from data generated in conjunction with the handover request to determine which party / multiple parties are responsible for which amount of the fees, and other example implementations.

[0086] Data generated in conjunction with requests to transfer remote valet parking services, and data generated for recording remote valet parking services provided to vehicles during a given journey, can be collected and maintained in a system (e.g., 605) for the remote valet parking service (e.g., 610) or in a cloud-based service (e.g., 150). This allows for the aggregation and crowdsourcing of the results of the remote valet parking services, thereby improving both the provision of future remote valet parking services and the autonomous driving models upon which vehicles rely for autonomous driving and requesting remote valet parking services, as well as other example uses.

[0087] Turn Figure 7Simplified block diagram 700 and simplified block diagram 1600 illustrate communication between systems during the delivery of an exemplary remote valet parking service. For example, a handover request 710 can be sent from a vehicle (105) (e.g., a remote valet parking support block (e.g., 705) of its autonomous driving system) via a network to a computing system that provides or acts as an agent for the remote valet parking service (provided through one or more remote value service systems (e.g., 605)). In other cases, a trusted third-party system (e.g., outside of autonomous vehicle 105) can determine that vehicle 105 requires assistance (e.g., through aggregate sensor data from various devices monitoring traffic involving the vehicle). In some cases, passengers within the vehicle may use a third-party service (e.g., cloud-based service 150) to trigger a remote valet parking service (e.g., via a smartphone app), which may send a handover request (e.g., 710') on behalf of vehicle 105, among other example implementations. A secure, high-priority communication channel 715 can be established between vehicle 105 and remote valet parking system 605 to enable the provision of remote valet parking services. For example, sensor data (e.g., camera data, lidar data, etc.) collected by sensors on vehicle 105 can be transmitted to provide a near real-time view of the vehicle's location and status, as well as its surroundings. In some cases, the data may include data from internal sensors of vehicle 105 (e.g., to provide a view of the vehicle's passengers and / or facilitate real-time communication between passengers and the virtual driver of the remote valet parking system), among other example uses. The virtual driver of the remote valet parking system can respond to information received describing the on-site conditions of vehicle 105 and use controls on their terminal to generate driving instruction data to be sent via channel 715 to remotely control the driving operations of vehicle 105. The remote valet parking service can also obtain supplementary data (e.g., in addition to the data received from vehicle 105) from external sources such as roadside units, other vehicles, drones, and other sensor devices. This information can be provided via a high-priority channel (e.g., 720) facilitated by one or more back-end systems (e.g., 150). In some implementations, the remote valet parking system 605 can determine a set of sensors (which can change dynamically as the vehicle moves along a path under the control of the remote valet parking driver) from the location of the vehicle 105. The remote valet parking system can establish another secure channel (e.g., 720) with the set of sensors and obtain real-time data describing the scene around the vehicle controlled by the remote valet parking system.Therefore, in some implementations, the remote valet parking service may use either or both of sensor data from sensors on the controlled vehicle 105 or sensor data from sensors outside the controlled vehicle 105.

[0088] As described above, in some implementations, the autonomous vehicle can detect instances where it should invoke a remote valet parking service for assistance. In some cases, this determination may be aided by one or more backend services (e.g., 150). In some implementations, the vehicle may provide such service 150 (or other cloud-based systems, storage, and services) with data describing the circumstances that facilitated the handover request (e.g., 710). The vehicle may further provide reports (after or during the service) describing the performance of the remote valet parking system (e.g., describing the maneuvers or routes taken by the remote valet, describing passenger satisfaction with the service, etc.). Such reporting data (e.g., 730) can later be used to train machine learning models and otherwise enhance the service provided by the backend or cloud-based system (e.g., 150). Insights and improved models may be derived by system 150 and then shared with the vehicle's autonomous driving system (and its remote valet parking support logic 705). In some cases, autonomous vehicles can record information describing the maneuvers and reactions of remote valet parking and use this information to further train and improve models used for their own autonomous driving machine learning models. Similarly, reported data (e.g., via 720) can be provided from the remote valet parking system 605 to cloud-based services or to the vehicle itself to enhance the vehicle's (and other vehicles') autonomous driving logic and handover requests, as well as other example uses such as those described herein.

[0089] As an illustrative example, an autonomous vehicle (e.g., 105) may autonomously determine (or based on passenger feedback or feedback received or reported by public safety officials, etc.) that its autonomous driving system is unable to handle specific situations while driving along a route. Therefore, a remote valet parking service can be triggered. In some cases, based on a prediction that the segment of road will be problematic, a remote valet parking service may be contacted before the upcoming segment. In some implementations, the handover request may be executed by a logic block that complements the autonomous driving system logic, which implements the path planning phase (such as in...) within the autonomous driving pipeline. Figure 5(As discussed in the examples). In some instances, once a remote valet handover request is issued to the remote valet parking system, a communication module on the autonomous vehicle (such as a Telematics Control Unit (TCU)) can be used to connect to the remote valet parking service. In some implementations, remote valet parking service communication can be established as communication with an emergency service (similar to an emergency call) specified during the TCU manufacturing phase. The vehicle's location can be provided in this handover request. In some implementations, the handover request and remote valet parking service can be implemented in a call / control center provided by the OEM, where a human virtual driver handling the "remote valet parking" takes action. In some implementations, in response to establishing a connection between the vehicle and the remote valet parking service, the remote valet parking can send a request to the vehicle to stream video from all its cameras to obtain a real-time view of the surrounding environment. Other sensors at the same location (e.g., road cameras and roadside sensors) can also be identified to provide data (e.g., additional streaming video) to supplement the information received from the vehicle. Remote valet parking relies on a real-time view of the vehicle's surroundings and road conditions, displayed to the valet via streaming video from the vehicle (and possibly from supplementary sources such as road cameras). This allows the valet to control the vehicle (similar to a video immersive game where players see a view of the car and control it using a steering wheel, handheld controller, etc.) to drive it to its destination. In some cases, the destination may correspond to the next segment of a route determined to be less problematic, in which case point control may be handed back to the autonomous driving system, allowing the vehicle to be driven in a standard autonomous driving mode. In other cases, based on the circumstances of the original handover request and detected characteristics, the remote valet parking service can guide the vehicle to a specific destination identified as having the capability to resolve problems detected at the vehicle, such as driving a vehicle with damaged sensors or an autonomous driving system to the nearest service center, or driving a vehicle with sick or injured passengers to a hospital, among other examples and use cases.

[0090] As described above, in some implementations, the autonomous driving system of a vehicle can access data collected by other remote sensor devices (e.g., other autonomous vehicles, drones, roadside units, weather monitors, etc.) to preemptively determine potential conditions on upcoming road segments. In some cases, individual sensors can provide data to a cloud-based system to aggregate and process this data set, thereby providing multiple autonomous vehicles with information about road segments and conditions affecting these routes. As described above, in some cases, the cloud-based system and other systems can receive inputs associated with previous pull-over and remote valet parking handover events and can detect common characteristics of these events. In some implementations, machine learning models can be built and trained based on this information, and such machine learning models can be deployed on and executed by roadside units, cloud-based support systems, remote valet parking computing systems, or the onboard systems of the autonomous vehicle itself to provide logic for predictively determining potential remote valet parking handovers. For example, the vehicle can preemptively identify areas along each road where frequent pullovers and / or remote valet handovers are common, using sensor data accessible by the autonomous vehicle. In some instances, the autonomous vehicle can (e.g., from a corresponding machine learning model) determine the likelihood of a pullover and / or remote valet handover (even if no pullover and handover have previously occurred at that particular road segment) based on reported conditions for an upcoming segment. Using this information, the autonomous vehicle can proactively prepare for a handover to the onboard driver or remote valet service. In some cases, the autonomous vehicle can decide to alter its route planning to avoid troublesome segments ahead (e.g., based on the detection of unavailable communication resources supporting remote valet parking, a lack of availability of reports for preferred valet services, user preferences to avoid remote valet parking as much as possible, etc.). In some implementations, the autonomous vehicle's display can warn or guide passengers about upcoming, predicted problems and the likelihood of a pullover and / or remote valet handover. In some cases, this information can be presented in an interactive display where passengers can register their preferences for handling the upcoming segment of their journey, i.e., by handing it over to a passenger, handing it over to a remote valet parking service, selecting an alternative route, or pulling over. In other implementations, cloud-based knowledge reflecting troublesome road segments can be passed to road signs or in-vehicle road maps to indicate the troublesome segment to the driver and other autonomous vehicles, among other example implementations.

[0091] Turn Figure 8Simplified block diagrams 800 and 1700 illustrate the collaborative reporting of information related to pullover event risks and road condition warnings. This information can be further utilized to initiate remote valet parking services to assist autonomous vehicles in navigating such dangerous and challenging scenarios. For example, sensor devices on the affected vehicle and / or in the surrounding area can collect information on pullover requests and / or remote valet parking events, and this information can be shared and utilized to enhance autonomous driving systems. Figure 8 In the example, when a pullover or handover occurs, the affected vehicle (e.g., 105) can assemble data generated and collected in association with the event and can share that information with cloud-based support systems (e.g., 150) and / or edge devices (such as roadside units or edge computer (or edge cloud) servers (e.g., 140)).

[0092] Figure 9 A simplified block diagram 900 illustrating the features of an example autonomous vehicle 105 is shown. The autonomous vehicle 105 may include various vehicle sensors (e.g., 620, 625), an AI / machine learning-based autonomous driving stack 515, and logic (e.g., 905) for supporting the triggering and generation of handover requests to a system capable of providing remote valet parking services. A telematics control unit (TCU) 910 may be provided, through which handover requests are sent and communication is established between the vehicle 105 and a virtual driver terminal providing remote valet parking services.

[0093] When the autonomous driving engine (e.g., 515) determines a pullover event or the remote valet parking support logic (e.g., 905) determines that a handover request should be sent, a signal can be sent to the TCU (910) to transmit the vehicle location and pullover location to the respective cloud-based entities (or a single entity or a gateway that distributes the information to multiple entities or services). In fact, many different services can utilize this type of information. For example, in one example, a cloud-based application 815 (e.g., associated with the vehicle OEM) might be the primary target or recipient of this information and can distribute portions of the information to other recipients. In other instances, the vehicle 105 can provide and distribute the data itself to multiple different cloud-based applications (e.g., one application per recipient). For example, an OEM maintenance application (e.g., 820) can utilize the pullover or handover information and use it to diagnose and identify extreme situations where the vehicle (and its models) is unable to handle autonomous driving. In some instances, recipients of pullover or handover information may include map application providers (e.g., 825, 826), including providers of traditional navigation maps, 3D maps, high-definition (HD) maps, etc., who may receive the information directly from the vehicle via a dedicated cloud application or through an OEM that receives the information directly from the vehicle. Map providers may use pullover and handover information for statistical purposes, which can help populate maps with information about areas prone to pullover incidents and challenging autonomous driving conditions, ensuring the information is continuously updated. Furthermore, HD maps may incorporate such information as part of the high-precision information provided for each road segment, among other examples. Municipalities, government agencies, toll road providers, and other infrastructure companies and management agencies (e.g., 830) may also be recipients of pullover and handover information (e.g., receiving pullover and handover information directly from vehicle 105, indirectly through another application or entity, or capturing such information through associated roadside sensors and roadside support units, among other examples). Such agencies can use this information to trigger road maintenance, as evidence of new road and infrastructure projects, policing, toll collection, to trigger the deployment of signs or warnings, and for other purposes.

[0094] Pulling over or handover events can also trigger information shared by vehicle 105 with nearby roadside units, vehicles, and other sensor devices. Example roadside units (e.g., 140) can utilize this information, for example, by processing the data along with other data it receives, and (e.g., via road segment application 835) sharing the information or its analysis results with other vehicles (e.g., 110) or systems in their vicinity. For example, the roadside unit can alert other vehicles to the risk of pull-over events, prepare infrastructure to support communication with remote valet parking services, and other example actions. The roadside unit can also store or transmit this information, making it accessible to relevant municipalities, maintenance providers, and agencies (e.g., to dynamically adjust traffic signal timing, update digital signage, open additional traffic lanes, etc.).

[0095] As mentioned above, various cloud-based and edge-based computing systems can leverage parking and handover information collected from various vehicles over time to improve models. These models can be shared and used to improve recommender systems (e.g., to recommend parking or remote valet handover), enable predictive or preemptive remote valet handover, improve autonomous driving models, improve remote valet services, and other example uses and benefits.

[0096] In various embodiments, mathematical models can be used (which guarantee safety if all road agents conform to the model, or correctly assign responsibility in the event of an accident). For example, a safety model could rely on mathematically calculated minimum longitudinal and lateral safe distances between two road agents to avoid a worst-case collision by modeling the behavior of the agents under a set of prescribed constraints.

[0097] Whenever the distance between two agents falls below the safe distance specified by the road safety model (e.g., a "hazardous situation"), the safety model can mathematically guarantee the prevention of a collision if both agents respond by setting accelerations within previously specified constraints (e.g., setting an "appropriate response"). On the other hand, if one of the agents is non-compliant, then that agent is liable in the event of an accident.

[0098] The use of a safety model simplifies the analysis of cases involving two agents by focusing on their longitudinal and lateral dimensions separately. For example, the agent’s velocity and acceleration, the minimum safe distance calculated using these velocities and accelerations, and the actual distance between the agents are all analyzed in terms of their longitudinal and lateral components in a coordinate system, where the center of the lane is assumed to be located on the y-axis (therefore, the longitudinal component is expressed in terms of y, while the lateral component is expressed in terms of x).

[0099] Figure 10 Various driving stages according to certain embodiments are depicted. Figure 10 In this model, agents 1002 and 1004 are depicted in three phases: 1006, 1008, and 1010. To conform to the safety model, when both the minimum safe distances in the longitudinal and lateral directions are violated, the agents need to formulate an appropriate response, and the appropriate response itself depends on which violation occurred most recently. In the first phase, 1006, agents 1002 and 1004 are separated by an unsafe lateral distance, but a safe longitudinal distance exists. The second phase, 1008, depicts the last point in time where the longitudinal distance remains safe (referred to as the "blame time"). At the next point in time after the blame time, the longitudinal safe distance is also violated. In the third phase, 1010, the agents have returned to a safe state after formulating an appropriate response in the longitudinal direction and have avoided a collision.

[0100] The safety model can be designed to be completely decoupled from the agent's policy. To conform to the safety model, the autonomous driving stack can include additional components to check the conformity of decisions made by the agent's policy and implement decisions that default to conforming to the safety model when the agent's policy requests actions that do not conform.

[0101] While safety models are designed with autonomous vehicles in mind, various embodiments of this disclosure include vehicles with control systems that use any suitable accident avoidance mathematical model as a mechanism for avoiding accidents decided by a human driver. Such embodiments may potentially result in greater overall safety for human drivers and may provide evidence or assurance that, where current laws allocate responsibility in a manner comparable to the liability allocation mechanism of the safety model, the driver will not be held liable for an accident (e.g., liability is assigned to an agent who violates model conditions). Following the safety model, the various embodiments described herein present another potential, longer-term advantage: for example, as more agents (human or other agents) are equipped with safety model implementers (or similar implementers), the total number of road accidents will decrease, moving towards an ideal situation for all agents.

[0102] In one particular embodiment of this disclosure, the vehicle includes a control system for replacing driver inputs that would result in accelerations that do not conform to the safety model with synthetically generated inputs that guarantee the generation of accelerations within an acceleration range conforming to the safety model. The driver input conforming to the safety model is transmitted to the actuation system without alteration, thereby enabling the system to take over only in potentially hazardous situations.

[0103] Figure 11A diagram depicts a system 1100 for modifying driver input to ensure acceleration conforms to a safety model, according to certain embodiments. In various embodiments, system 1100 may be part of a vehicle (e.g., 105), and any module shown may be implemented by any suitable logic of the vehicle's (e.g., 105) computing system. In other embodiments, any module may be implemented outside the vehicle (e.g., by 140 or 150), and the results may be passed to the vehicle. System 1100 includes a control unit 1102 (in various embodiments, control unit 1102 may have any suitable characteristics of a drive control unit 220), a sensor suite 1104 (in various embodiments, sensor suite 1104 may have any suitable characteristics of sensor 225), a safety model 1106, a safety model implementer 1108, a control-accelerometer 1110, and an acceleration-control converter 1112. In certain embodiments, all components of system 1100 may be integrated within the vehicle. In other embodiments, one or more components may be different from the vehicle and communicatively coupled to the vehicle.

[0104] A control device 1102 may be provided to enable a human driver to provide input to the vehicle's actuation system. For example, the control device may include a steering wheel or other steering mechanism, an accelerator pedal or other throttle, and a brake pedal or other braking mechanism. In one embodiment, the control device may include other components such as a gearshift lever, emergency brake, joystick, touchscreen, gesture recognition system, or other suitable input control devices that may affect the speed or direction of the vehicle.

[0105] Sensor suite 1104 may include any suitable combination of one or more sensors utilized by the vehicle to collect information relating to the world state associated with the vehicle. For example, sensor suite 1104 may include one or more lidar, radar, cameras, global positioning systems (GPS), inertial measurement units (IMUs), audio sensors, infrared sensors, or other sensors described herein. World state information may include any suitable information, such as any situation described herein, objects detected by the sensors, location information associated with the objects, or any other suitable information.

[0106] World state can be provided to any suitable component of system 1100, such as safety model 1106, control-accelerometer 1110, or acceleration-control converter 1112. For example, world state information can be provided to safety model 1106. Safety model 1106 can use the world state information to determine the range of accelerations conforming to the safety model for a vehicle. In doing so, safety model 1106 can track the longitudinal and latitudinal distances between the vehicle and other vehicles or objects. Additionally, safety model 1106 can also track the longitudinal and latitudinal velocities of the vehicle. Safety model 1106 can periodically update the range of accelerations conforming to the safety model and provide the acceleration range to safety model implementer 1108. The acceleration conforming to the safety model can specify the range of accelerations conforming to the safety model in the longitudinal direction and the range of accelerations conforming to the safety model in the latitudinal direction. Acceleration can be expressed in any suitable unit, such as the square of meters per second, and can have positive or negative values ​​(or can be zero).

[0107] Safety model implementer 1108 receives control signals from driver input and invokes control-accelerometer 1110. If the driver input is passed to actuation system 1114 (in some embodiments, this actuation system includes both lateral and longitudinal acceleration components), the converter converts the driver input into an acceleration value indicating the predicted vehicle acceleration. Safety model implementer 1108 can determine whether the acceleration value is within the range of the latest safety model-compliant acceleration received from safety model 1106. If the acceleration value is within the range of safety model-compliant acceleration, the safety model implementer allows driver input from control unit 1102 to be passed to actuation system 1114. If the acceleration value is not within the range of safety model-compliant acceleration, the safety model implementer blocks driver input and selects a safety model-compliant acceleration value within the received range. Safety model implementer 1108 can then invoke acceleration-control converter 1112 with the converted acceleration value and can receive one or more control signals in return. In certain embodiments, the control signals provided by the acceleration-control converter 1112 may have the same format as the control signals provided to the actuation system 1114 in response to driver input. For example, the control signals may specify braking amount, acceleration amount and / or steering amount and steering direction, or other suitable control signals. The safety model implementer 1108 may provide these new control signals to the actuation system 1114, which may use the control signals to accelerate the vehicle as specified.

[0108] In various embodiments, the safety model implementer 1108 can select any suitable acceleration value within the range of accelerations conforming to the safety model. In one particular embodiment, the safety model implementer 1108 can randomly select an acceleration value from this range. In another implementation, the safety model implementer 1108 can select the most conservative or least conservative value from this range. In yet another embodiment, the safety model implementer 1108 can select a value in the middle of the range. In yet another embodiment, the safety model implementer 1108 can use strategy information (e.g., based on driver preferences or based on safety considerations) to determine the acceleration value. For example, the safety model implementer 1108 can favor longitudinal acceleration over latitudinal acceleration, and vice versa. As another example, the safety model implementer 1108 can favor acceleration that is more comfortable for the driver (e.g., slower braking or smaller steering adjustments may be better than emergency braking or steering). In various embodiments, decisions can be based on safety and comfort, as well as relevant metrics calculated from the same set of motion parameters and vehicle characteristics.

[0109] As described above, the control-accelerometer 1110 converts driver input (e.g., steering wheel rotation and accelerator / brake pedal pressure) into acceleration. In various implementations, the converter 1110 can consider any suitable information during the conversion, such as world conditions (e.g., vehicle speed, weather, road conditions, road layout, etc.) and the physical properties of the controlling vehicle (e.g., vehicle weight, vehicle shape, tire properties, braking properties, etc.). In one implementation, the conversion can be based on a complex mathematical model of vehicle dynamics (such as one supplied by the vehicle manufacturer). In some embodiments, the converter 1110 can implement a machine learning model (e.g., any suitable regression model) to perform the conversion. Figure 12 and Figure 13 A more detailed description of the example machine learning model used for control-acceleration conversion.

[0110] Acceleration-control converter 1112 may include logic for converting a safety-model compliant acceleration, implemented by safety-model actuator 1108 during takeover, into an input suitable for actuation system 1114. Converter 1112 may utilize any suitable information to perform this conversion. For example, converter 1112 may utilize any one or more fragments of information used by control-acceleration converter 1110. Similarly, converter 1112 may use methods similar to those of converter 1110, such as a machine learning model suitable for outputting a control signal given an input with acceleration. In a particular embodiment, the acceleration-control converter may include a proportional-integral-derivative (PID) controller for determining the desired control signal based on the acceleration value. The PID controller may be implemented using a classical controller algorithm with proportional, integral, and derivative coefficients, or it may be based on machine learning, where ML algorithms (e.g., implemented by machine learning engine 232) are used to predict these coefficients, utilizing optimization metrics that take into account safety and comfort.

[0111] 1114 can refer to any suitable actuation system for receiving one or more control signals and causing a vehicle to respond to one or more control signals. For example, an actuation system may adjust the amount of gasoline or electricity (or other power source) supplied to the engine or motor of the vehicle, the amount of braking pressure applied to the wheels of the vehicle, the amount of angle applied to one or more wheels of the vehicle, or make any other suitable adjustment that may affect the acceleration of the vehicle.

[0112] Figure 12 A training phase of a control-accelerometer 1110 according to certain embodiments is depicted. Training input 1202 for the model may include any suitable information that could affect the acceleration determined in response to a control signal. For example, training input may include any combination of the vehicle's initial speed, road conditions, tire conditions, weather conditions, steering wheel rotation, accelerator pedal pressure level, brake pedal pressure level, road layout, vehicle physical properties, or other suitable information, and the acceleration obtained under each set of such information. Such data may be used during a machine learning training phase 1204 to train a regression model 1206, which may be used by the vehicle to convert control signals and other information (e.g., world state information, vehicle physical properties) into acceleration values. In various embodiments, regression model 1206 is trained on basic fact data collected from one or more vehicles of a vehicle category under many different weather, road, and vehicle state conditions. In various embodiments, training may be performed by any suitable computing system, whether in an onboard computing system, a cloud-based system, or other data processing environment.

[0113] Figure 13 An inference phase of a control-accelerometer 1110 according to certain embodiments is depicted. In the inference phase, various inputs 1302 associated with the vehicle are provided to a regression model 1206, which outputs predicted accelerations based on the inputs. The inputs may reflect the types of inputs used to train model 1206, but may include real-time values ​​of such inputs. The regression model 1206 outputs acceleration values ​​1304.

[0114] A similar regression model can be used for the acceleration-control converter 1112. Similar input data can be used to train the model, but during inference, the model can receive the desired acceleration (along with real-time values ​​of the world state and / or vehicle state) as input and can output a control signal predicted to cause the desired acceleration.

[0115] Figure 14 A flow diagram, according to certain embodiments, for providing acceptable control signals to a vehicle actuation system is described. At 1402, a first set of one or more control signals is generated in response to human input to the vehicle. At 1404, a determination is made as to whether the first set of control signals will result in an acceptable acceleration of the vehicle. If the control signals will result in an acceptable acceleration, the control signals are provided to the vehicle actuation system unchanged at 1406. If the control signals will result in an unacceptable acceleration, the acceptable acceleration is identified at 1408. At 1410, the acceptable acceleration is converted into a second set of one or more control signals. At 1412, the second set of one or more control signals is provided to the vehicle actuation system in place of the first set of one or more control signals.

[0116] Safely transferring driving responsibility from autonomous vehicles to humans, or vice versa, is a critical task. As mentioned above, one approach to this transfer can be based on methods such as safety models, where the autonomous vehicle can intercept unacceptable human input and replace it with safer input.

[0117] In various embodiments of this disclosure, handover readiness can be based on a measurement of the overall signal quality of the vehicle's sensors relative to the context in which such measurements are taking place. This context can be any suitable context described herein, such as traffic conditions (e.g., a highway or busy street) or weather conditions (e.g., sunny, rainy, presence of puddles, presence of thin ice, etc.). The signal quality metric can be determined using a machine learning (ML) algorithm that takes sensor data and contextual information as input and outputs a signal quality metric. This signal quality metric is then used to determine handover readiness using another ML algorithm trained with vehicle collision information. If the signal quality metric indicates poor signal quality based on the context, the handover from a human driver to an autonomous vehicle may not be permitted because such a handover could be unsafe.

[0118] Figure 15 Training phases for constructing a context model 1508 according to certain embodiments are described. In various embodiments, the context model 1508 may be a classification model constructed using sensor data 1504 and contextual information base facts 1506. ML algorithm 1502 may represent any suitable algorithm for training the context model 1506 based on sensor data 1508 and contextual information base facts 1504. Sensor data 1504 may include any suitable sensor data from one or more sensors of a vehicle, such as one or more lidar, radar, cameras, global positioning systems (GPS), inertial measurement units (IMUs), audio sensors, infrared sensors, or other sensors. ML algorithm 1502 may train the context model 1506 using various instances of sensor data 1508 and contextual information base facts 1504, wherein each instance may include a set of sensor data and an associated context. In various embodiments, training data may include actual sensor data and associated context, simulated data and associated context, and / or (e.g., synthetic data from synthetic images generated using the methods described herein) and associated context. In a particular embodiment, a context may include one or more textual keywords describing the context (such as “foggy” and “wet road”), but any suitable contextual expression is contemplated in this disclosure.

[0119] Figure 16Training phases for constructing a signal quality metric model 1608 according to certain embodiments are described. In various embodiments, the signal quality metric model 1608 may be a regression model constructed using sensor data and contextual information basis facts. In various embodiments, sensor data 1604 may be the same sensor data as sensor data 1504, or may be sensor data that is at least partially different from sensor data 1504. In some embodiments, contextual information basis facts 1606 may be contextual information that is the same as contextual information basis facts 1506, or may be contextual information that is at least partially different from contextual information basis facts 1506. ML algorithm 1602 may train the signal quality metric model 1608 using various instances of sensor data 1604 and contextual information basis facts 1606, wherein each instance may include a set of sensor data and an associated context. In various embodiments, training data may include actual sensor data and associated context, simulated data and associated context, and / or synthetic data and associated context. By analyzing multiple distinct instances of sensor data associated with a specific context, the ML algorithm 1602 may be able to train a signal quality metric model 1608 to distinguish between the quality of individual instances of sensor data 1604 within a specific context. Similar training can be performed for any suitable number of distinct contexts.

[0120] After the signal quality metric model is trained, it can receive instances of sensor data (wherein instances of sensor data include sensor data collected over a period of time) and the associated context, and output one or more indicators of sensor data quality. For example, a signal quality metric may include a comprehensive score of the quality of instances of sensor data. In another example, a signal quality metric may include a score for the quality of each of multiple types of sensor data. For example, a signal quality metric may include scores for camera data and scores for LiDAR data. In some embodiments, the score may be any of a variety of quality metrics, such as a measurement of signal-to-noise ratio, a measurement of resolution, or other suitable types of quality metrics. In some embodiments, a signal quality metric may include scores for multiple types of quality metrics or may include a single score based on multiple types of quality metrics. In some embodiments, the score of a signal quality metric may be a normalized value (e.g., from 0 to 1).

[0121] Figure 17 The training phase for constructing a handover readiness model 1708 according to certain embodiments is described. In various embodiments, the handover readiness model 1708 may be a classification model constructed using signal quality metric information 1704 and collision information basis facts 1706.

[0122] ML algorithm 1702 can represent any suitable algorithm for training a handover readiness model 1708 based on signal quality metric 1704 and collision information basis facts 1706. ML algorithm 1702 can train scenario model 1508 using individual instances of signal quality metric 1704 and collision information basis facts 1706. Instances used for training may include the signal quality metric and a set of collision information. The set of collision information may include any suitable safety outcome associated with a specific instance of the signal quality metric. For example, an instance of collision information may indicate whether an accident occurred when the autonomous vehicle was operated under the signal quality metric. As another example, an instance of collision information may indicate whether an accident was nearly caused when the autonomous vehicle was operated under the signal quality metric. As yet another example, an instance of collision information may indicate whether an accident occurred or was nearly caused when the autonomous vehicle was operated under the signal quality metric (e.g., a near-accident could be considered the same as an actual accident). In various embodiments, training data may include real data signal quality metrics and collision information, simulated data signal quality metrics and collision information, synthetic data signal quality metrics and collision information, or a combination of the foregoing.

[0123] Figure 18 An inference phase for determining a handover decision 1808 based on sensor data 1802, according to certain embodiments, is described. In this inference phase, for example, which may be implemented by an onboard computing system during driving time, sensor data 1802 is collected and provided to a trained scenario model 1508. The scenario model analyzes sensor data 1504 and determines scenario 1804 from sensor data 1802. The determined scenario 1804, together with sensor data 1802, is provided to a signal quality metric model 1608. The signal quality metric model 1608 analyzes sensor data 1802 and scenario 1804 and determines a signal quality metric 1806 based on the quality of sensor data 1802 in scenario 1804. The signal quality metric 1806 is provided to a handover readiness model 1708, which determines the handover decision 1808 based on it. In a particular embodiment, the handover decision 1808 is a binary indication of whether the handover is safe or unsafe. In other embodiments, this may be a multi-class decision with three or more possible outcomes. For example, handover decisions can include any number of outcomes, each representing a different safety range for the handover. In various embodiments, the vehicle can use the handover decision 1808 outcome to determine whether to perform a handover or not, or to perform a partial handover, such as handing over some control but not others (e.g., handing over steering but not braking, or vice versa).

[0124] In various embodiments, the inference phase may be executed periodically or in response to a trigger (or both periodically and in response to a trigger). For example, while the autonomous vehicle is exercising driving control, the inference phase may be executed periodically to determine whether the autonomous vehicle is still reliably capable of exercising driving control. As another example, the inference phase may be triggered upon receiving a request to transfer control from a human driver to the vehicle. As yet another example, the inference phase may be triggered by a change in context or a significant change in the quality of sensor data.

[0125] In certain embodiments, handover is preemptively planned based on known static data levels, such as the availability of high-resolution maps of the roads the vehicle will travel. This type of data may be unavailable for certain areas in which the vehicle must drive, for example, because high-resolution map data for a certain area has not yet been collected. In such cases, the system can preemptively plan the handover (e.g., before the trip begins) and use any of the handover techniques described herein to prepare the driver in advance for a safe handover. In a particular example, the inference phase for determining the handover decision is triggered when the vehicle enters an area without high-resolution map data (or just before entering). In some embodiments, the availability of high-resolution map data can be used as input to signal quality metric model 1608 to positively influence the signal quality metric when high-resolution map data is available, or negatively influence the signal quality metric when high-resolution map data is unavailable. In some embodiments, the high-resolution map is essentially considered as an additional sensor input.

[0126] In various embodiments, references Figures 15-18 The described ML algorithm or model can be trained or executed by any suitable computing system (such as an in-vehicle computing system, a support system implemented using cloud-based and / or fog-based computing resources) or in another data processing environment.

[0127] Figure 19 A process for determining whether to transfer control of a vehicle, according to certain embodiments, is described. At 1902, the vehicle's computing system determines a signal quality metric based on sensor data and the context of that data. At 1904, the likelihood of safety associated with the transfer of control of the vehicle is determined based on the signal quality metric. At 1906, the transfer is prevented or initiated based on the likelihood of safety.

[0128] Autonomous vehicles are expected to offer potential advantages over human drivers in terms of better and more consistent responses to driving events, due to their immunity to factors that negatively impact humans, such as fatigue, varying levels of alertness, emotional fluctuations, or other factors. However, autonomous vehicles may be affected by equipment malfunctions or may experience situations where they are not adequately prepared to operate (e.g., an autonomous vehicle may enter an area with new features that the vehicle's algorithm was not trained on), thus necessitating handover to a human driver or pulling the vehicle over.

[0129] In various embodiments of this disclosure, the driver's condition (e.g., level of fatigue, alertness, emotional state, or other status) is analyzed before the vehicle is handed over to a human driver to improve the safety of the handover process. Abruptly handing over control to someone who is not prepared may prove more dangerous than not handing over at all, as demonstrated by several recent accidents using recently tested vehicles.

[0130] Typically, autonomous vehicles have outward-facing sensors because the perception system focuses on mapping the environment, while the positioning system focuses on finding the location of the autonomous vehicle based on data from these sensors and the map data. Various embodiments of this disclosure provide one or more onboard cameras or other sensors to track the driver's state.

[0131] Figure 20 The training phase of a driver state model 2008 according to certain embodiments is described. During the training phase, sensor data 2004 and driver state basic fact data 2006 are provided to an ML algorithm 2002, which trains the driver state model 2008 based on this data. In various embodiments, the driver state model 2008 may be a classification model that outputs categories describing the driver's state. In other embodiments, the driver state model 2008 may be a regression model that outputs scores of the driver's state (where higher scores depict a more desired state).

[0132] In various embodiments, sensor data 2004 may represent any suitable sensor data and / or information derived from sensor data. For example, sensor data 2004 may include or be based on image data collected from one or more cameras capturing images of the interior of a vehicle. In some embodiments, one or more cameras or a computing system coupled to the cameras may implement AI algorithms to detect facial, eyebrow, or eye movements and extract features to track fatigue levels and alertness levels indicated by the detected features.

[0133] In various embodiments, sensor data 2004 may include or be based on one or more temperature maps collected from an infrared camera. In some embodiments, the infrared camera or a computing system coupled to the infrared camera may implement AI algorithms to track the driver's emotional state or other physical state based on these temperature maps. As merely an example, a rise in a human driver's body temperature (e.g., indicated by an increase in the number of red areas on a temperature map) may indicate an agitated state. In various embodiments, sensor data 2004 may include or be based on pressure data collected from tactile or haptic sensors on the steering wheel, accelerator, or driver's seat. In some embodiments, a computing system coupled to such tactile or haptic sensors may implement AI algorithms to analyze such pressure data to track the driver's alertness level or other physical state.

[0134] In various embodiments, sensor data 2004 may include or be based on electrocardiogram (EKG) or inertial measurement unit (IMU) data from wearable devices, such as smartwatches or health tracker bands. The computing system coupled to such wearable devices or the wearable devices themselves may utilize AI algorithms to extract EKG features to track the driver's health or other physical conditions, or analyze IMU data to extract features to track the driver's alertness level or other physical conditions.

[0135] In various embodiments, sensor data 2004 may include or be based on audio data from in-cabin microphones. Such data may be preprocessed using noise cancellation techniques to isolate sounds generated by passengers in the vehicle. For example, if the in-vehicle infotainment system is playing audio, the signal from the playing audio can be subtracted from the audio captured by the in-cabin microphones before any further processing. The raw audio features may be used directly to measure a user's responsiveness or overall physical state (e.g., slurred speech may indicate intoxication), but can also be used to classify audio events (e.g., laughter, crying, yawning, snoring, gagging, or other events) that can be used as further features indicative of the driver's state. The analyzed audio data may also include speech detected from conversations between passengers or between passengers and the vehicle's infotainment system (e.g., speech may be converted to text by an automatic speech recognition engine, etc.). As an example, in addition to communicating with the driver about the handover, the vehicle's dialogue system may attempt to obtain the driver's confirmation of the upcoming handover. Speech can be transformed into text and then analyzed by a sophisticated natural language processing pipeline (or similar pipeline) to classify the speaker's intent (e.g., positive or negative confirmation), analyze the sentiment of the interaction (e.g., negative sentiment towards the corpus (such as profanity)), or model the topics of discussion. Such outputs can then be used as additional features for driver state tracking algorithms.

[0136] Features related to the state of the vehicle can also provide insights into the driver's current level of alertness. As examples, such features may include one or more of the following: media currently playing in the vehicle (e.g., movies, video games, music), cabin light levels, the amount of driver interaction with dashboard controls, window aperture levels, the state of the cabin temperature control system (e.g., air conditioning or heating), the state of devices connected to the vehicle (e.g., a cellular phone connected via Bluetooth), or other vehicle state inputs. Such features may be included in sensor data 2004 as input to the ML algorithm 2002 to train the driver state model 2008.

[0137] In certain embodiments, activity labels can be derived from sensor data by an activity classification model. For example, the model can detect whether the driver is sleeping (e.g., based on closed eyes in image data, snoring heard in audio data, and decreased body temperature), arguing with another passenger in the cabin (e.g., increased volume, rapid heartbeat, mutual insults), feeling unwell (e.g., gagging sounds captured by a microphone and the driver's head bent downwards as shown in image data), or any other suitable activity.

[0138] In various embodiments, raw sensor data may be fed into training algorithm 2002. Additionally or alternatively, classification based on the raw sensor data may be fed into ML algorithm 2002 to train driver state model 2008. In some embodiments, the aforementioned activity labels may be fed into training algorithm 2002 (optionally, lower-level features and / or raw sensor data may also be fed into training algorithm 2002) to obtain more robust driver state tracking results.

[0139] The driver state basic facts 2006 may include known driver states corresponding to instances of sensor data 2004. When the driver state model 2008 implements a classification algorithm, the driver state basic facts 2006 may include various categories of driver states. When the driver state model 2008 implements a regression algorithm, each instance of the driver state basic facts 2006 may include a numerical score indicating the driver state.

[0140] In various embodiments, the driver state basic facts 2006 and sensor data 2004 may be driver-specific or may include data aggregated for multiple different drivers.

[0141] Figure 21The training phase of the handover decision model 2110 is described. The ML training algorithm 2102 uses driver history data 2104, driver state 2106, and handover decision basis facts 2108 to train the handover decision model 2110. In an alternative embodiment, the ML algorithm 2102 may simply use driver state 2106 and handover decision basis facts 2108 to train the handover decision model 2110. The handover decision basis facts 2108 may include actual previous handover decisions and their corresponding outcomes (e.g., whether a collision or other hazardous event occurred). In certain embodiments, all or a subset of the handover decision basis facts 2108 may be simulated to augment the dataset.

[0142] Driver history data 2104 may include any suitable background information that can inform the driver's level of attention. For example, history data 2104 may include the driver's historical data, including: instances of drunk driving (DUI), past accidents, instances of potentially dangerous actions taken by the driver (e.g., swerving into oncoming traffic, slamming on the brakes to avoid rear-ending another vehicle, running over a speed bump), the driver's health condition, or other suitable background information. In some embodiments, the autonomous vehicle may have a driver ID slot into which the driver inserts a unique ID, and the autonomous vehicle connectivity system retrieves the driver's relevant historical data. The driver's background information may be obtained in any other suitable manner.

[0143] In the described embodiment, during the training phase, the driver's historical data 2104, along with driver state information 2106, is fed to the ML algorithm 2102 to construct a handover decision model 2110 that outputs two or more categories. In one embodiment, the handover decision model 2110 outputs three categories: handover, no handover, or short-term handover. In another embodiment, the handover decision model 2110 outputs two categories: handover or no handover. In yet another embodiment, one of these categories may be partial handover. As examples, the "handover" category may indicate that a handover can be performed with a high level of confidence, while the "no handover" category may indicate a lower level of confidence, where continued control by the vehicle is undesirable, potentially leading to a delay in handover until a remote monitoring system takes over control of the vehicle until the driver is ready or the vehicle is brought to a safe stopping point. The "short-term handover" category may indicate a moderate level of confidence in the driver and, in some embodiments, may lead to a handover of control to the driver with a time limit, within which the vehicle is forced to stop (e.g., the vehicle may be brought to a safe stopping point by a backup unit, such as a communication system that can control the vehicle or provide a storage location for the vehicle). In another embodiment, "partial handover" may indicate a moderate level of confidence in the driver and may lead to only partial control being transferred to the driver (e.g., only braking control or only steering control). In one embodiment, "conditional handover" may indicate a moderate level of confidence in the driver and may lead to a handover to the driver and monitoring of the driver's actions and / or the user's status to ensure the vehicle is being operated safely. The above are merely examples of possible handover categories, and the handover decision model 2110 can output any combination of the above handover categories or other suitable handover categories.

[0144] In various embodiments, the context detected by outward-facing sensors of the vehicle may also be taken into account to assess the driver's ability to successfully handle the handover. For example, weather conditions, visibility conditions, road conditions, traffic conditions, or other conditions may affect the level of alertness required for the handover. For instance, if these conditions are adverse, a different level of alertness may be required before the handover to the driver. This can be achieved by feeding contextual information into machine learning algorithm 2102 or in any other suitable manner.

[0145] Figure 22The inference phase for determining handover decision 2208 according to certain embodiments is depicted. Sensor data 2202 as described above is provided to driver state model 2008, which outputs driver state 2204. Driver state 2204 and historical data 2206 are provided to handover decision model 2110, which outputs handover decision 2208 as described above or other suitable handover decision. In other embodiments, the handover decision model may consider other factors (e.g., the context of driving conditions determined from one or more outward sensors) or omit historical data 2206.

[0146] The inference phase can be executed in response to any suitable trigger. For example, the inference phase can be executed in response to determining that the vehicle cannot operate independently at an acceptable level of safety. As another example, the inference phase can be executed periodically while a human driver is operating the vehicle, and the result of the inference phase can be a determination of whether the driver is fit to operate the vehicle. If the driver is unfit, the vehicle can take over all or part of the driving control, can provide a warning to the driver, or can take actions to increase the driver's alertness (e.g., playing loud music, opening windows, vibrating the driver's seat or steering wheel, or other appropriate actions).

[0147] When the system determines that a handover is to be made to a human driver, the driver is notified of the impending handover. To do this, the system can engage with the driver in one or more of several possible ways. For example, the system can engage with the driver verbally. For instance, text with correct semantics and grammar can be constructed by a natural language generation engine and then transformed into synthesized speech audio by a text-to-speech engine to produce spoken information describing the handover. As another example, the system can engage with the driver physically. For instance, a motor mounted on the driver's seat or steering wheel can cause the seat or steering wheel to vibrate violently, while considering driver safety so as not to startle the driver and cause an accident. In other embodiments, the system can engage with the driver in any suitable manner to transfer the handover.

[0148] Figure 23 A process for generating a handover decision, according to certain embodiments, is described. At 2302, sensor data is collected from at least one sensor located inside the vehicle. At 2304, the sensor data is analyzed to determine the physical condition of a person inside the vehicle. At 2306, a handover decision is generated, at least in part, based on the person's physical condition, indicating whether the person is expected to be able to safely operate the vehicle.

[0149] As discussed in this paper, some autonomous driving systems can be equipped with features that support the transfer of control from the autonomous vehicle to a human user inside the vehicle or at a remote location (e.g., in remote valet parking applications). In some implementations, the autonomous driving system can employ a logic-based framework to smoothly transfer control from the passenger (EGO) to the autonomous (agent) vehicle and vice versa in different situations and circumstances, with the aim of enhancing the safety of both the passenger and the road. At least some aspects of this framework can be parallelized (e.g., via FPGA, Hadoop clusters, etc.) to be implemented on the hardware of the autonomous driving system.

[0150] For example, the example framework could consider different scenarios where it is safer for either the autonomous vehicle or the human driver to regain control of the vehicle, and suggest mechanisms for realizing these control requests between the two parties. As an example, there might be situations where the autonomous vehicle might want to regain control of the vehicle for safer driving. The autonomous vehicle could be equipped with cameras or other internal sensors (e.g., microphones) that could sense the driver's state of mind (e.g., determining if the driver is distracted by a phone call or feeling drowsy / drowsy) and determine whether to take over control based on the driver's level of awareness. The autonomous vehicle could include mechanisms for analyzing sensor data (e.g., analyzing data from cameras and microphones inside the vehicle) and requesting and taking over control from the driver if the driver's level of awareness is low or the driver is otherwise deemed unsafe (e.g., driving under the influence, driving with hands off the wheel, sleeping while driving, texting while driving, reckless driving, etc.), or if the autonomous vehicle senses any unusual activity in the vehicle (e.g., fighting, screaming, or other unsafe driving behavior by the human driver or passengers). In this way, the safety of both those inside and outside the autonomous vehicle can be enhanced.

[0151] In some implementations, authentication-based (e.g., using biometrics) command control can be used to prevent unauthorized use of autonomous vehicles. As an example, in some embodiments, when an autonomous vehicle is stolen or falls into the wrong hands, it may be able to detect the situation and lock itself to prevent control. For example, the autonomous vehicle may include an authentication mechanism that uses biometrics (e.g., fingerprints, voice and facial recognition, driver's licenses, etc.) to authenticate the user requesting control of the autonomous vehicle. These mechanisms can prevent unauthorized use of the autonomous vehicle. In some cases, the use of the autonomous vehicle or aspects thereof may be provided based on different levels of permission. For example, one user may be able to fully manually control the car anywhere, while another user may only be able to control the car at specific geofenced locations. As another example, in some embodiments, passengers may request control of the autonomous vehicle when certain situations arise, such as very congested roads, inclement weather, damaged sensors (e.g., cameras, lidar, radar, etc.). In response to a request, the autonomous vehicle can authenticate the user based on one or more of the user's biometrics, and if authenticated, control of the autonomous vehicle can be transferred to the user. As another example, in some embodiments, when an entity / user (e.g., law enforcement, first responders, government officials, etc.) wishes to remotely control the autonomous vehicle, the autonomous vehicle can authenticate the user before transferring control to the entity / user.

[0152] In some embodiments, such as instances where surrounding autonomous vehicles perceive that the autonomous vehicle is driving dangerously or outside the acceptable range of behavior models of other vehicles, control of the autonomous vehicle can be crowdsourced to multiple surrounding vehicles (including law enforcement vehicles) or infrastructure-based sensors / controllers. In such instances, the entity / multiple entities requesting control can be authenticated, for example, by biometrics of the person requesting control or by digital security information (e.g., digital certificates) from the autonomous vehicle / infrastructure sensors.

[0153] Figure 24 The figure illustrates a high-level block diagram of the above-described framework according to at least one embodiment. For example, in scenario 2402, when an autonomous vehicle (e.g., via camera or microphone data from inside the autonomous vehicle) detects an unsafe driving condition (e.g., Figure 24 In situations listed below (or other unsafe conditions), where the autonomous vehicle is operating in human-driven / manual operation mode, control will be restored to the autonomous vehicle to continue driving in autonomous mode. In this scenario, the autonomous vehicle may request the driver to regain control of the vehicle before regaining control.

[0154] In scenario 2404, the human driver responds to situations where the driver identifies a condition in which the driver feels uncomfortable continuing in autonomous operation mode (e.g., Figure 24 (Those listed in the text) and request control of the autonomous vehicle. The autonomous vehicle may initiate an authentication request at 2405, for example, using biometrics or other authentication methods, to authenticate the human driver in response, and upon successful authentication, transfer control from the autonomous vehicle to the human driver (otherwise, the autonomous vehicle will retain control).

[0155] In scenario 2406, for example, due to observed unsafe driving of the autonomous vehicle, due to a report of the autonomous vehicle being stolen, due to the need to move the autonomous vehicle for crowd / road control purposes, etc., law enforcement officials or (multiple) adjacent autonomous vehicles may request control of the autonomous vehicle. The autonomous vehicle may initiate an authentication request at 2407, in response to which the requesting person / entity is authenticated, and upon successful authentication, control may be transferred from the autonomous vehicle to the official / adjacent (multiple) autonomous vehicles (otherwise, the autonomous vehicle retains control).

[0156] Figure 25 This is a diagram illustrating an example process for controlling the takeover of an autonomous vehicle according to at least one embodiment. Operations in the example process can be performed by aspects or components of the autonomous vehicle. Example process 2500 may include additional or different operations and may be performed in the order shown or in another order. In some cases, Figure 25 One or more of the operations shown are implemented as procedures comprising multiple operations, subroutines, or other types of routines. In some cases, operations may be combined, executed in a different order, executed in parallel, iterated, or otherwise repeated or executed in another manner.

[0157] At point 2502, the autonomous vehicle operates in autonomous mode, thereby controlling many or all aspects of its operation.

[0158] At point 2504, the autonomous vehicle receives a request from another entity to take over control of the autonomous vehicle. This entity may include a human passenger / driver of the autonomous vehicle, a person remote from the autonomous vehicle (e.g., law enforcement or government officials), or another autonomous vehicle or multiple autonomous vehicles in the vicinity of the autonomous vehicle (e.g., crowdsourced control).

[0159] At point 2506, the autonomous vehicle prompts the entity to provide credentials to authenticate the entity requesting control. This prompt may include biometric prompts, such as fingerprints, voice samples for speech recognition, facial samples for facial recognition, or another type of biometric. The prompt may also include prompts for other types of credentials, such as usernames, passwords, etc.

[0160] At 2508, the autonomous vehicle receives input from the requesting entity, and at 2510, determines whether the entity is authenticated based on the received input. If the entity is authenticated, the autonomous vehicle allows takeover and transfers control to the requesting entity at 2512. If the entity is not authenticated based on the input, the autonomous vehicle rejects the takeover request at 2514 and continues operating in autonomous mode.

[0161] Figure 26 This is a diagram illustrating another example process for controlling the takeover of an autonomous vehicle according to at least one embodiment. Operations in the example process can be performed by aspects or components of the autonomous vehicle. Example process 2600 may include additional or different operations and may be performed in the order shown or in another order. In some cases, Figure 26 One or more of the operations shown are implemented as procedures comprising multiple operations, subroutines, or other types of routines. In some cases, operations may be combined, executed in a different order, executed in parallel, iterated, or otherwise repeated or executed in another manner.

[0162] At 2602, the autonomous vehicle is operated in a manual / human-driven mode, wherein a human (inside or away from the autonomous vehicle) controls one or more aspects of the operation of the autonomous vehicle.

[0163] At 2604, the autonomous vehicle receives sensor data from one or more sensors located inside the vehicle, and at 2606 analyzes the sensor data to determine whether the input from the human operator is safe. If the input is determined to be safe, the autonomous vehicle continues to operate in manual operation mode. If the input is determined to be unsafe, the autonomous vehicle requests to take over control from the human operator at 2608 and operates in autonomous operation mode at 2610.

[0164] The transition from Level 2 (“L2” or “L2+”) autonomous vehicles to Level 5 (“L5”) autonomous vehicles with full autonomy could take several years, and the autonomous vehicle industry may observe a gradual shift in responsibility from the human-driver role until full autonomy (driverless) is achieved everywhere and in every situation. During this transition, achieving safe takeover from machine control (autonomous mode) to human control (human driving mode) is crucial, but this comes with several challenges. One potential challenge is controlling random interventions from human drivers that occur without a request from the autonomous system. Another challenge arises from event-driven interventions. Three types of takeovers that can occur in autonomous vehicles include:

[0165] Vehicle-requested takeover: This occurs when a vehicle requests driver takeover and transitions from autonomous mode to human driving mode. This can happen in some situations, such as when an autonomous vehicle faces new circumstances with its perception systems (e.g., when there is some uncertainty regarding optimal decision-making, or when the vehicle leaves a geofenced area). A common method for requesting human takeover is to alert the driver through one or more means, such as a pop-up message on the dashboard, a beep, or a vibration on the steering wheel. When a human driver adapts to takeover, some errors may occur during the takeover due to longer reaction times than expected, lack of focus, or other reasons.

[0166] Random takeover by a human driver: Possible takeovers can occur randomly by a human driver (e.g., without a request from the vehicle) and for unpredictable reasons. For example, a human driver might be distracted or unexpectedly awakened from sleep, reacting inappropriately (quickly taking control of the steering wheel without full awareness). As another example, a human driver might be in a rush (e.g., catching a flight or attending an important event), dissatisfied with the vehicle's speed in autonomous mode, and therefore might take over control to accelerate. These types of random takeovers can be undesirable because developing driving rules / strategies to address such unpredictable takeovers is impractical, and random takeovers themselves can lead to accidents / collisions.

[0167] Event-Driven Human Takeover: Another possible takeover can occur due to unpredictable events. For example, a human driver might suddenly feel the need to get out of the car (e.g., due to claustrophobia, discomfort, etc.). As another example, a passenger traveling with a human driver might suddenly enter a high-risk situation, and the human driver might take over to bring the car to a stop. As yet another example, a human driver might feel uncomfortable on the road they are traveling on (e.g., dark and unknown roads), triggering a need to take control to feel more comfortable. These types of takeovers can be undesirable because they disrupt autonomous driving in an unpredictable way, and the takeover itself could lead to an accident / collision. Similar to the previous cases, this type of takeover is also undesirable because it is impractical to develop driving rules / strategies for such unpredictable takeovers, and takeovers driven by unpredictable events cannot be safe.

[0168] Of these types, random and event-driven takeovers may be considered unsafe, and accordingly, autonomous driving systems may be specifically configured to detect and control these types of takeovers, which could allow for safer driving and avoid unpredictable behavior during autonomous driving modes. In some embodiments, to mitigate these potentially unsafe takeover situations: • (For example, as implemented in an in-vehicle perception software stack) The autonomous driving perception phase can be extended to include software modules for real-time detection of unsafe takeovers; • The autonomous driving action phase (e.g., vehicle control software and hardware implemented in the vehicle system) can be extended to include software modules for mitigating detected unsafe takeovers in real time. • The autonomous driving planning phase (e.g., (multiple) route planning subsystems) can be extended to include mitigation measures to include considering potential changes to the autonomous driving mode, routes, or other adjustments to avoid discomfort for passengers or the driver.

[0169] Figure 27 This is an illustration of an example autonomous driving pipeline 2800 for sensing, planning, and acting in autonomous vehicles according to at least one embodiment. Specifically, Figure 27 This paper provides an overview of some considerations in the perception and control of autonomous vehicles to detect and mitigate potential unsafe takeovers in real time. The operation of the perception, planning, and action execution pipeline can be performed by the onboard control system of the autonomous vehicle. As shown, the example perception, planning, and action execution pipeline includes a sensing / perception phase, a planning phase, and an action / control execution phase.

[0170] In the example shown, the control system receives sensor data from multiple sensors coupled to the autonomous vehicle, including vehicle perception sensors (e.g., multiple cameras, lidar, etc.) and vehicle control elements (e.g., steering wheel sensors, brake / accelerator pedal sensors, multiple interior cameras, interior microphones, etc.). The control system uses the sensor data from the sensing / perception phase to detect unsafe takeover requests from the human driver of the autonomous vehicle. Detection of unsafe takeover can be based on at least a portion of the received sensor data. For example, unsafe takeover can be detected based on sensors coupled to the accelerator pedal, brake pedal, and / or steering wheel to sense the takeover behavior. In some cases, in-vehicle cameras and / or multiple microphones (e.g., utilizing artificial intelligence) can be used to detect multiple actions by the driver indicating a desire to take over control of the autonomous vehicle. In some embodiments, data from the pedal / steering wheel sensors and onboard cameras can be correlated to detect potential human takeover requests and determine whether these actions constitute an actual or unintentional takeover request. For example, a driver who is suddenly awakened or distracted may actuate one or more of the following: the brakes, the accelerator, or the steering wheel, without intending to initiate a random takeover of control.

[0171] Upon detecting that a requested takeover is unsafe, the control system mitigates the unsafe takeover request. This may include, for example, blocking the takeover request, making it possible that a human driver is not allowed to control the autonomous vehicle. For instance, during autonomous driving mode, the steering wheel, brake actuators / pedals, and accelerator actuators / pedals may be locked and can only be unlocked if the autonomous vehicle requests human takeover (this could be in response to detecting that a random takeover request is safe, as described below). Furthermore, the doors may remain locked in response to an unsafe takeover request, because in some cases, door unlocking may only be enabled when the vehicle is parked (not moving).

[0172] In some cases, mitigating unsafe takeover requests may involve modifying the autonomous driving mode to match the driver's / passenger's wishes. For example, the control system may reroute the autonomous vehicle's route (e.g., direction, speed, etc.) to ensure driver / passenger comfort and minimize the risks to passengers / drivers introduced by the takeover request. In some cases, the control system may respond to a takeover request by prompting human driver and / or passenger input (e.g., using voice prompts (for autonomous vehicles with voice recognition enabled) and / or text prompts), and may modify one or more aspects of the autonomous mode based on the input received from the driver / passenger.

[0173] Figure 28This is a diagram illustrating an example process of a human driver requesting takeover control of an autonomous vehicle according to at least one embodiment. Specifically, Figure 28 The diagram illustrates an unsafe takeover detection and mitigation scheme. Operations in the example process can be performed by components of the autonomous vehicle (e.g., the autonomous vehicle's control system). Example process 2800 may include additional or different operations and may be performed in the order shown or in another order. In some cases, Figure 28 One or more of the operations shown are implemented as procedures comprising multiple operations, subroutines, or other types of routines. In some cases, operations may be combined, executed in a different order, executed in parallel, iterated, or otherwise repeated or executed in another manner.

[0174] At 2802, the autonomous vehicle operates in autonomous driving mode. For example, the autonomous vehicle's control system may be controlling one or more aspects of its operation (e.g., through perception, planning, and a motion pipeline). At 2804, the autonomous vehicle (e.g., based on sensor data passed to the control system) determines whether it has encountered an unconventional or unknown situation. If so, at 2806, the autonomous vehicle requests a human driver to take over control, and at 2808, the autonomous vehicle enters and operates in human driving mode (where the human driver controls the autonomous vehicle). The autonomous vehicle can then determine at 2810 whether it encountered a normal / known situation during human driving mode. If so, the autonomous vehicle can request a takeover of control or regain control at 2812 and re-enter autonomous operation mode. If no unconventional / unknown situation was encountered at 2804, the autonomous vehicle continues to operate in autonomous driving mode, thus allowing it to continuously determine whether it has encountered unconventional / unknown situations.

[0175] At 2814, the autonomous vehicle detects a takeover request from a human driver. The takeover request may be based on sensor data from one or more sensors coupled to the autonomous vehicle, wherein the sensors may include sensors located inside the autonomous vehicle (e.g., sensors coupled to the steering wheel, brake actuators, accelerator actuators, or (multiple) internal cameras or (multiple) microphones).

[0176] At point 2816, the autonomous vehicle determines whether the takeover request is unsafe. If so, in response, the autonomous vehicle can mitigate the unsafe takeover request. For example, at point 2818, the autonomous vehicle can block the takeover request. Alternatively, at point 2818, the autonomous vehicle can prompt the driver for input (e.g., using voice recognition software to enable dialogue with the driver) to learn more about the reasons for the takeover request or any unusual circumstances.

[0177] At point 2820, the autonomous vehicle determines the driver's situation or the reason for initiating a takeover request based on input received from the driver. If, for example, the situation is identified as a risk to the driver or passengers (e.g., screaming, unsafe behavior, etc.), then route replanning may need to be considered, and therefore the autonomous vehicle can modify its autonomous driving mode at point 2822 to pull over. If, for example, the situation is identified as uncomfortable for the driver and / or passengers (e.g., unknown route / road, very dark environment, etc.), then the autonomous vehicle can modify its autonomous driving mode to provide the driver / passengers with more visual information that can be displayed at point 2824 (e.g., displaying (additional) route details); the autonomous vehicle can also adjust the interior lighting to allow the driver to see additional information to help the driver and / or passengers achieve greater comfort with the autonomous driving mode. If, for example, the situation is identified as a complaint about speed (e.g., the driver wants the autonomous vehicle to slow down or accelerate), then an alternative speed and / or route can be considered during the planning phase, and the autonomous vehicle can modify its autonomous driving mode to change the speed (or route). Other mitigation strategies may be adopted in response to received driver input.

[0178] One of the potential benefits of autonomous vehicles is the possibility of a much safer driving environment. However, despite the best efforts to create error-free automated systems, mechanical, physical, and / or electronic damage caused by wear and tear on vehicles is unavoidable. Such damage can lead to malfunctions in autonomous vehicles.

[0179] Inevitably, when autonomous vehicles are damaged, specifically involving damage to their sensors, the vehicle's functionality will be diminished. For example... Figure 29 As shown, the level of automation of autonomous vehicles is defined relative to the amount of human driver involvement required. When an autonomous vehicle encounters a problem, a human passenger (or a remote monitoring entity) may need to take over driving control, or the vehicle may cease operation.

[0180] Furthermore, the likelihood of an accident increases when a vehicle malfunctions—whether it's a sensor problem, a processor or memory failure, or any other hardware / software issue. This is also true if a human driver is forced to take over control of the vehicle, especially if that driver is not prepared to do so. The ability to track what is happening on a vehicle can prove invaluable to many parties. For example, insurance companies, drivers, or vehicle manufacturers can benefit from various liability issues. Additionally, vehicle designers can benefit from understanding what happens in critical situations.

[0181] Figure 30 The figure illustrates a comprehensive cognitive supervision system 3000. System 3000 is a computational system configured with logic (such as a subsystem or implementation of the computational system discussed herein) to supervise and adjust the autonomous vehicle's level of autonomy based on continuous analysis of the driving status and accuracy of the autonomous vehicle (specifically, the sensing layer, planning layer, and action layer of the autonomous vehicle). System 3000 may include multi-level intelligent mechanisms for handling potential problems with the autonomous vehicle by monitoring, alerting, and re-engaging the human driver and safely handing over driving control to the human driver. System 3000 may also be configured to allow remote supervision and / or control of the autonomous vehicle. System 3000 can also be considered as a system for reducing the autonomous vehicle's level of autonomy, thereby relying more on the human driver in the event of sensor or component failure of the vehicle or other situations that the vehicle cannot handle.

[0182] System 3000 can monitor the autonomy level of autonomous vehicles. Furthermore, the system can determine whether the autonomy level is correct, and if incorrect, can change the vehicle's autonomy level. Additionally, if a change is necessary, System 3000 can alert the driver. The system can also alert the remote monitoring system 3010 to the change.

[0183] The Integrated Cognitive Supervision System (C2S2) 3005 can reside on top of the conventional automation systems of the autonomous vehicle (e.g., it can supervise the conventional automation systems of the autonomous vehicle). In one example, system 3005 resides on top of the vehicle's sensor system (3020), planning system (3030), and execution system (3040). It should be noted that in some implementations, C2S2 can reside on top of or co-operate with additional onboard computing systems of the autonomous vehicle. Specifically, C2S2 can reside on top of any system that may affect the vehicle's level of autonomy. System 3005 can also record the history of autonomous driving levels and sensor health monitoring. The collected data can be very concise and accessible offline, allowing for reference in the event of any malfunction or accident.

[0184] In some examples, C2S2 3005 includes executable logic for monitoring the level of autonomy in a vehicle and comprises three main modules: functional assurance, quality assurance, and safety assurance. Each of these main modules can have a set of predefined key performance indicators (KPIs) for accepting or rejecting the current state of autonomy set for the vehicle. If C2S2 determines that the level of autonomy is unacceptable due to any of the monitored modules, C2S2 can have the ability to change the level of autonomy of the vehicle. Furthermore, the system will notify the human driver of this change. The ability to change the level of autonomy can be very beneficial. For example, if some sensor malfunction occurs, C2S2 can determine that the level of autonomy can be reduced, rather than completely disabling, the vehicle's autonomy, rather than completely removing it. This could mean the vehicle moving from Level 4 to Level 3 (e.g., as...). Figure 31 (As depicted in the document). Such changes may not require human driver involvement in controlling the vehicle, but in some embodiments, the change in autonomy can be passed to the driver to allow the driver to pay closer attention when he or she is needed.

[0185] continue Figure 30 For example, C2S2 3005 will evaluate the KPIs of each of the three main blocks (functional assurance, quality assurance, and safety assurance) of the three systems (sensors 3020, planning 3030, and execution 3040). If C2S2 3005 detects any problems in the system, it can assess whether a change in the level of autonomy is necessary. Not every problem will require a change in the level of autonomy. For example, one of the sensors in a vehicle may be malfunctioning. However, if that sensor produces duplicate data relative to another sensor, the vehicle may not lose its ability to maintain its current level of autonomy.

[0186] However, in other examples, sensor issues can lead to problems. Even when manufacturers have introduced specific vehicles capable of achieving Level 4 autonomy, such designations are conditional in practice, and the vehicle's autonomy may change over time. For example, the level of autonomy may have to be changed when sensors malfunction or passenger safety is compromised in scenarios such as sensor / component failure. C2S2 3005 allows for changes to the level of autonomy and notifies both the driver and the remote monitoring system (3010).

[0187] In addition to monitoring and changing autonomy levels, the C2S2 3005 can also report actions back to the remote monitoring system 3010. Not only can the C2S2 3005 report changes in autonomy levels, but it can also report any significant data to the remote system 3010. For example, in cases where a necessary change in autonomy level is necessary, or even in the event of an accident involving autonomous vehicles, a complete record of the level change and data related to vehicle movement, planning, autonomy levels, etc., can be sent to and stored by the monitoring system 3010. Such data is useful in identifying faults in accidents and providing data for improvement. It is conceivable that any data that can be captured can be sent to the remote monitoring system 3010 if needed.

[0188] Figure 30 The system depicted herein represents only modules that may appear in a particular embodiment. Other embodiments may include additional modules not specifically mentioned herein. Furthermore, not every module is necessary, or modules may be combined in other embodiments.

[0189] While providing a completely human-free driving experience using autonomous vehicles may be ideal, some human driver interaction may be necessary during vehicle operation, depending on the level of autonomy of the vehicle. This is especially true in emergency situations, where a human driver taking over control may be necessary. In such cases, a typical handover to a human driver, if successful, can take an average of about three seconds. However, humans are often inattentive, easily distracted, and tend to react slowly to certain situations. Therefore, maintaining driver involvement while the vehicle operates at an autonomous node to achieve a quick and safe handover can be challenging.

[0190] Accordingly, at least in some cases, human backup in handover scenarios within autonomous vehicles may be unreliable. If personnel cannot react quickly enough, potentially dangerous situations can be exacerbated by inattentive drivers who fail to react in time. The various implementations of the systems described above can provide a safer method for handover between autonomous and human drivers.

[0191] Figure 32 The diagram illustrates an example of the architectural data flow for an autonomous vehicle operating at Level 4 autonomy. Figure 32 An example flow includes a sensing module 3210, a planning module 3220, an action module 3230, and a wired driver (“DBW”) module 3240. As an example, the sensing module 3210 can be responsible for processing data from various sensing sensors (e.g., camera, radar, lidar, GPS, etc.). The sensing module 3210 can have any suitable characteristics of sensor 225. Data output from the sensing module (which may represent the vehicle's motion parameters, such as speed, position, and direction) and data representing objects around the vehicle can be passed to the planning module 3220 (which may have any suitable characteristics of a path planner module (e.g., 242), such as those discussed elsewhere herein). The planning module 3220 can make relevant decisions about actions to be taken while driving on the road based on the current situation. The decisions made by the planning module can be passed to the action module 3230, which may include a controller for generating specific vehicle commands to be given to the DBW module 3240. These commands may include, for example, commands for a specific steering angle and / or acceleration. These commands are then executed by the DBW module. It should be noted that the above process is merely exemplary and other processes may exist. In addition, it is possible for different vehicles to have different levels of intelligence. For example, a vehicle rated L2 will have a different level of intelligence than a vehicle rated L4.

[0192] Currently, Figure 32 In the event of a failure at one of the module levels in the example, or if the vehicle's planning algorithm is unable to take action in certain driving scenarios, the vehicle will automatically send a signal to the driver indicating that the driver needs to take over. This signal can be visual, audio, or a combination thereof. Figure 33 The image shows an example of sending a video signal to the driver.

[0193] Figure 34 The diagram illustrates the handover process for an example autonomous vehicle. As you can see, at the beginning of the process, the vehicle may be in autonomous mode at point 3410. Once a problem is detected and the autonomy level needs to be changed, a takeover signal is sent at point 3420. Finally, the autonomous mode is deactivated at point 3430.

[0194] A non-abrupt and spontaneous handover process will help the driver engage the vehicle when necessary. Furthermore, if sensor malfunctions occur, it may not be necessary to make the vehicle completely non-autonomous. Simply reducing the level of autonomy may be safe. For example, for an autonomous vehicle operating in L4 mode, it may be unnecessary to have the vehicle directly hand over to a human driver and shut down its autonomy. The planning algorithm (e.g., performed by planning module 3220) relies on multiple sensor inputs. The reliability of the autonomous system is defined by the accuracy with which the planning algorithm can make decisions based on these sensor inputs. Each system has its set of critical and non-critical sensor inputs, which defines the confidence level of the decisions made by the planning module. If a subset of the sensors of an L4-level vehicle (primarily redundant sensors) ceases to operate, the L4-level vehicle can no longer operate at the same confidence level. In the example case, the vehicle may simply degrade from an L4 confidence level to an L3 confidence level, requiring more attention from the driver. However, complete driver takeover and shutting down the vehicle's autonomous system may be unnecessary.

[0195] Figure 35 The diagram illustrates an example of the process of transferring control of an autonomous vehicle to a human driver. Additionally, Figure 35 The diagram illustrates the coordination between human responses and the actions of autonomous vehicles. This coordination is represented by dashed lines. Figure 35 An example process can be performed in the autonomous transportation planning module 3220. However, it should be noted that... Figure 35 The process can be executed by any module or combination of modules of the computing system, including those not mentioned in this article.

[0196] Figure 35 The initial example (3510) shows an autonomous vehicle operating normally in its autonomous mode (Level 4 in this example). Therefore, the human driver is inactive (3515). This may be especially true for higher levels of autonomy in autonomous vehicles.

[0197] When a problem occurs, the vehicle can issue a system malfunction alarm (3520). The human driver will then receive this alarm (3525). This alarm can be visual, audio, policy-based, or any other type of alert.

[0198] If the malfunction is determined not to be severe enough to require immediate driver interaction, the vehicle can switch to a lower autonomous mode (3530). In this example, the vehicle switches from L4 to L3. The human driver will become aware of this transition accordingly (e.g., based on an alert received at 3525), and may pay attention to the driving situation, and may gain control of the vehicle for a certain period of time if needed (3535). In some examples, the vehicle can confirm driver involvement by using certain sensors and monitoring. For example, the vehicle may use gaze monitoring, haptic feedback, audio feedback, etc.

[0199] If another error occurs, the vehicle can issue another system malfunction alarm (3540). Once again, the driver will receive the alarm (3545) after it has been sent.

[0200] Next, if it is determined once again that the level of autonomy can be lowered (from L3 to L2 in this example), the vehicle will lower its level of autonomy again (3550). Now, in the corresponding action, the driver begins to pay closer attention (3555). In this example, the human driver will pay constant attention because the car is in L2 mode.

[0201] If the vehicle needs to lower its autonomy level again, this time to L1, the driver will need to take over. Therefore, the vehicle can issue a takeover signal (3560). In the corresponding action, the driver can receive the takeover signal (3570).

[0202] Now, the vehicle can confirm whether the driver will be able to take over control of the vehicle. Therefore, the vehicle will wait for the driver to gain control (3562). As mentioned earlier, in addition to monitoring whether the driver has actually gained control, the vehicle can use monitoring and sensors to determine the driver's readiness status.

[0203] After a certain period of time, if the vehicle determines that the driver has not gained control (or cannot safely gain control), the emergency system is activated (3564). This may include performing different actions depending on the situation. For example, it may be necessary for the vehicle to pull over to the side of the road. In some cases, pulling over may not be safe, so the vehicle may continue driving for a period of time. Therefore, the vehicle may slow down and / or pull over to the side of the road until it is safe to stop. Once the emergency system is activated, the status of the emergency action will be completed accordingly (3574).

[0204] However, if the driver is able to take over and the handover is successful, the autonomous mode can be deactivated (3566). In the corresponding action, the driver will be fully involved and drive the vehicle (3576). It can be seen that early warnings (multiple warnings before a necessary handover) allow the driver to prepare for the handover before system failures and when the driver must take over.

[0205] Depending on the level of autonomy of the autonomous vehicle, some human driver interaction may be necessary while the vehicle is in operation. Even when the vehicle is generally capable of operating fully autonomously, there may be situations where it is necessary for a human driver to take over control (e.g., in an emergency). In other situations, driver takeover of control of the autonomous vehicle may be desirable, for example, when the human driver has a desire to drive or when there is a favorable reason for human control of the vehicle. However, humans are often inattentive, easily distracted, and slow to respond to certain situations. Accordingly, at least in some cases, human backup may be unreliable in handover situations within autonomous vehicles. Furthermore, human response time and reaction can vary depending on the situation. For example, some people have slower reaction times than others. As another example, some people react calmly in emergencies, while others panic.

[0206] Handover systems for autonomous vehicles and processes for implementing personalized methods of transferring control of the vehicle to a human can be beneficial. Such systems and processes can enhance the safety and effectiveness of handovers. This is particularly true for Level 5 autonomous vehicles, where a human driver is generally not required. In some cases, the human driver may be asleep or distracted, increasing the risks associated with the handover. When planning handovers, personalized and coordinated approaches can take into account the human driver's level of attention and / or reaction characteristics in such situations.

[0207] In various embodiments, the approach to personalizing and coordinating handovers can be applied to both planned and unplanned handovers in autonomous vehicles. While full autonomy may be desirable, real-world scenarios (such as, for example, critical sensor failures, unexpected and sudden changes in road conditions (e.g., flash floods)) may present situations beyond the capabilities of autonomous vehicles.

[0208] According to embodiments described herein, solutions to the handover problem may include a multi-pronged approach that considers one or more of the following when planning the handover: driver activity, individual capabilities, and the target route. This approach allows systems (e.g., onboard processing system 210) to make better judgments about whether and when to hand over control of the vehicle to a human driver. Furthermore, various embodiments may provide driver personalization over time and can continuously maintain contextual information references to progressively improve the handover process.

[0209] Figure 36 The figure illustrates an example system 3600 for handing over autonomous vehicles to human drivers. This system can also be considered a system for safe, timely, and adaptive handover from autonomous vehicles to human drivers. In some embodiments, the various modules may be implemented by an onboard processing system (e.g., 210). In other embodiments, one or more modules may be implemented by a cloud-based computing system (e.g., 140 or 150). System 3600 includes an occupant activity monitoring module 3610, a personalized occupant capability database 3615, a general occupant capability database 3620, a handover prediction module 3625, a handover handling module 3630, and an execution evaluation and optimization module 3635. System 3600 is merely exemplary and is not limited to the embodiments specifically presented herein.

[0210] Occupant Activity Monitoring (“OAM”) module 3610 extracts information related to the human driver of the autonomous vehicle. In certain embodiments, OAM module 3610 implements a combination of rule-based machine learning methods and deep learning methods. OAM can determine state characteristics associated with the human driver, such as the direction the driver is facing (e.g., whether the person's seat faces the road or the rear of the vehicle), the driver's seat positioning (e.g., the distance between the driver's seat and the steering wheel, the seat back angle, or any other characteristic of the driver's seat relative to the steering wheel), whether the driver is awake or asleep, whether the driver is engaged in another activity (e.g., reading, watching videos, playing games, etc.), or other state characteristics. The determinations made by OAM module 3610 and listed herein are merely exemplary, and OAM can be used to determine any characteristics of the driver that are considered to be related to the driver's ability to take full or partial control of the vehicle.

[0211] The OAM module 3610 can use data from several different sensors as input. For example, onboard sensors that can provide information to the OAM module 3610 include, for instance, cameras, inertial measurement units, seat and backrest sensors, ultrasonic sensors, or biometric sensors (e.g., heart rate monitors, body temperature monitors, etc.). Data from the sensors can be in raw format or can be preprocessed. The sensors listed herein are merely exemplary, and any type of sensor (either listed herein or not) can be used as data input to the OAM module 3610.

[0212] The General Occupant Capability (“GOC”) database 3620 may include data relating to statistics about the characteristics of general drivers similar to those of the actual drivers of the autonomous vehicle. For example, the GOC database 3620 may contain information about the characteristic responses of drivers with characteristics similar to those of the actual drivers (e.g., gender, age, physical health level). Furthermore, the information stored in the database 3620 may be global or specific to one or more geographic regions. In some embodiments, the GOC database 3620 may be external to the vehicle and become available to the autonomous vehicle via the cloud. The GOC database 3620 may be updated at any suitable time or interval, allowing handover operations of the autonomous vehicle to be improved over time. It should be noted that the GOC database 3620 may include more than one database.

[0213] Examples of data types in the GOC database can include the amount of time a driver with similar characteristics (e.g., individuals with similar characteristics, such as age, gender, etc.) spends performing actions such as: responding to cues, rotating the seat 180 degrees, moving the seat longitudinally a certain distance, placing his or her hands on the steering wheel from his or her legs, engaging in road activities upon leaving (which may depend on the driver's activities prior to being prompted to engage), or other appropriate activities associated with the handover. Additionally, driver characteristics (e.g., the driver's health condition) can be used to generate statistics corresponding to the context of the driver's condition. For example, the database can capture information indicating that a typical driver with the same lower back problems as drivers of autonomous vehicles might spend an average amount of time adjusting the seat to an upright position or moving the seat forward toward the steering wheel.

[0214] In addition to utilizing available statistical data, machine learning models, such as those implemented by the onboard processing system 210, can also be used to process raw data from autonomous vehicles. In other embodiments, such machine learning models can run in the cloud (rather than locally on the autonomous vehicle), and the inferred outputs can be utilized on the vehicle.

[0215] The Personalized Occupant Capability (“POC”) database 3615 may contain data similar in nature to the GOC database 3620. However, the POC database includes driver and vehicle-specific information, rather than information aggregated from multiple drivers as in the GOC database. Data in the POC database 3615 can help improve the functionality of the system 3600 because each person will differ from the baseline established by the GOC database 3620. Data in the POC database 3615 can be observed and measured over time. The POC module 3600 of the system 3615 can be considered the central location for maintaining and tracking the differences between the driver and the hypothetical driver.

[0216] In addition to driver-specific information, the POC database 3615 may also contain vehicle-specific information. For example, the time it takes for a driver to turn the seat 180 degrees may depend on the vehicle's technological capabilities, and the driver cannot speed up or slow down the process.

[0217] As examples, the following types of data can be stored in a POC database: the time it takes for a driver to respond to audio cues is X1 seconds longer than that of a typical driver; the time it takes for a driver to turn their seat is X2 seconds shorter than that of a typical driver (e.g., this could be due to the vehicle having rapid steering maneuvers and / or the driver's relatively quick response); the time it takes for a driver to move their seat longitudinally is X3 seconds slower than that of a typical driver; the time it takes for a driver to place their hands on the steering wheel is X4 seconds faster than that of a typical driver; and the time it takes for a driver to be fully conscious on the road is X5 seconds faster than that of a typical driver. While these examples discuss measurements relative to a typical driver, in some embodiments, the information stored in the POC database may include absolute measurements (e.g., the average time it takes for a driver to respond to audio cues is Y1 seconds, the average time it takes for a driver to turn their seat is Y2 seconds, etc.). Additionally, similar to a GOC database, a POC database can include other driver characteristics that can be used to generate statistics that can provide more contextual information about the driver's situation. As an example, a POC database may include information indicating how quickly a driver will move to bring their seat to an upright position or move their seat forward due to a back injury.

[0218] The Handover Prediction (HOF) module 3625 determines when a handover may be necessary. The HOF module can consider road conditions (such as, for example, accidents, overcrowded roads, public events, pedestrians, construction, etc.) to determine where and when a handover from an autonomous driver to a human driver may be required. The HOF module can receive, for example, local map and route information with real-time updates on traffic, accidents, hazards, and road maintenance. Some or all of this information can be stored locally within the autonomous vehicle or in the cloud (the vehicle can receive updates to this information via the cloud).

[0219] Figure 37 The diagram illustrates an example route 3705 taken by vehicle 3705 from point A to point B. Route 3705 has been selected by the vehicle 3705's navigation system (e.g., route planner module 242). The HOF module 3625 can take into account road conditions (such as accidents, overcrowded sections, and road construction sites) to determine where a handover to a human driver may be necessary. Figure 37 In the example, three areas (3710, 3720, and 3730) have been identified as potentially requiring such a handover. After identifying the possible handover locations, the HOF module 3625 can rank these locations based on different criteria. Examples of such criteria may include:

[0220] 1. Are there alternative routes that may be less preferred but do not require handover to a human driver at the identified location? As an example, location 3720 could be associated with alternative route 3715.

[0221] 2 – Can autonomous vehicles handle the identified handover locations by reducing speed and / or intermittently stopping when necessary? As an example, location 3730 has ongoing road construction, which could potentially slow traffic in a controlled and safe manner.

[0222] 3 – Are there any sections of the route that autonomous vehicles would be unable to handle without human intervention? For example, location 3710 might be such a location because an accident could cause severe traffic disruption. Autonomous vehicles need to ensure that human drivers are prepared in advance when approaching that particular location.

[0223] In various embodiments, the HOF module 3625 can determine handover locations along the route and rate their relative importance (e.g., which handover locations are more likely to require a handover).

[0224] Return to Figure 36The handover handling (“HOH”) module 3630 can consider handover-related information to make a handover decision. The HOH module 3630 accepts the outputs of the OAM module 3610, the POC database 3615, the GOC database 3620, and the HOF module 3625, and makes a handover decision based on one or more of these outputs.

[0225] Finally, the Execution Evaluation and Optimization (“EAO”) module 3635 compares the expected outcome of the handover with the driver’s actual behavior. The results of the comparison are fed back to the POC database 3615 and the HOH module 3630 for future handover improvements. To gather information, the EAO module 3635 may use the following example criteria for each handover event along the route: how long it took the driver to respond to the handover request; whether the driver was within the expected steering range after the handover; whether the driver’s acceleration / deceleration was within the expected acceleration / deceleration range after the handover; and how long it took the driver to immediately get back on the road after the handover. The criteria listed here are merely exemplary, and not all criteria are used in the various embodiments, or unlisted criteria may be used.

[0226] Updates within the POC database 3615 allow the handover process to incorporate more personalized information based on the technical capabilities of the driver and the autonomous vehicle. Thus, over time, as the number of autonomous vehicle rides increases, the POC database 3615 begins to differentiate itself more significantly from the GOC database 3620.

[0227] The HOH module 3630 uses feedback from the EAO module 3635 to calculate when and where the driver exhibits anomalies that deviate from typical behavior. This may differ from the content stored for drivers in the POC database 3615, as it relates to deviations from expected driver behavior and may be considered in future handovers. If the HOH module 3630 takes such anomalies into account in future handovers, road safety can be improved because the autonomous vehicle will make handover decisions and handover performance assessments based on data that is more representative of both the driver and the autonomous vehicle, as this data is based on real-world observations.

[0228] Figure 38 The diagram illustrates an example of the advanced operating procedure 3800 of the HOH module 3630. This operating procedure can also be considered a method for handing over autonomous vehicles to human drivers. Initially, the method begins by obtaining the determined handover locations from the HOF module 3625. These can be determined from the calculated priorities of the route.

[0229] Next, method 3800 continues to retrieve general driving data from GOC database 3620. It should be noted that obtaining any general driving data may be unnecessary. For example, when a sufficient amount of data is stored in POC database 3615, data from GOC database 3620 can be omitted from certain determinations. It should also be noted that personalized data may be transferred from one location to another; for example, when a driver purchases a new vehicle, information may be transferred from the old vehicle or the cloud to the new vehicle.

[0230] After obtaining general data (if utilized), the HOH module continues to obtain personalized data from the POC database 3615. It should be noted that there may be situations where personalized data is unavailable, such as, for example, when the vehicle is brand new and data has not yet been obtained.

[0231] Next, method 3800 continues to obtain data from OAM module 3610. This data may include information related to the driver, as it relates to the driver's attention level, activity, etc.

[0232] The HOH module can then determine the expected driver action for each of the possible handover locations determined by the HOF module 3625. If the HOH module 3630 determines it is time for a handover, the driver is prompted. If not, the HOH module 3630 can determine if there are any real-time updates from any other modules. If so, one or more updates can be used to re-determine the expected driver action.

[0233] continue Figure 38 For example, after the driver is prompted, the HOH module 3630 will determine whether the driver is capable of taking over. If so, the HOH module can provide the expected driver behavior to the EAO module 3635, and then can transfer control of the vehicle to the driver. Furthermore, data on how the driver behaved during the handover can be obtained from the EAO module 3635, and the expected driver behavior can be updated in response. The driver can continue driving, for example, until the driver is ready to relinquish control or makes a determination that the AV can safely regain control.

[0234] If the driver is not ready to take over when prompted, the HOH module 3630 can assess whether there are alternatives to the handover. This could include, for example, taking an alternative route, slowing down, etc. If an alternative exists, it can be selected. If no alternative would allow the vehicle to continue moving, the HOH module can guide the vehicle to a stop. It should be noted that this may involve altering the desired route to ensure the vehicle stops in a safe location and in a safe manner. If the vehicle stops, it can remain stationary until the driver is ready to take over. The driver can then drive until he or she is ready to allow the autonomous vehicle to take over again and it is safe to do so. In other embodiments, it is also possible for the vehicle to remain at a stop until an alternative exists that would allow the vehicle to move again. This could include, for example, a change in road conditions that would cause the vehicle to stop, or even a new route that has been opened.

[0235] The following example utilizes Figure 37 Example operation process and Figure 38 The operation of the HOH module 3630 will be illustrated by combining route examples.

[0236] Before the journey:

[0237] 1. The optimal route (3700) between A and B has been calculated and provided to the HOF module.

[0238] 2. HOF module 3625 uses real-time updates to identify three junction areas (3710, 3720, and 3730).

[0239] 3. The HOF module determines that location 3710 is the most likely place to require driver support (because there is very little information about accidents at that point).

[0240] 4. Location 3730 was selected as the next most likely location, where another handover may be required due to ongoing road construction.

[0241] 5. Location 3720 is identified as another potential junction area where increased pedestrian traffic is observed on the road. Autonomous vehicles can easily handle this section of the route by taking alternative route 3715 without driver assistance.

[0242] 6. The GOC database 3620 provides general information about the driver to the HOH module 3630.

[0243] 7. The POC database is empty (because the driver has just bought a car, which has limited personalized information based on the driver).

[0244] 8. The OAM module 3610 confirms that the driver is sitting behind the steering wheel and his son is sitting in the back seat.

[0245] 9. The model of the vehicle the driver is driving has a fully rotatable driver's seat to allow him to freely interact with the passengers in the back while driving. Thus, the driver has his back to the road and begins to talk to his son.

[0246] 10. The in-cabin camera provides comprehensive coverage of what is happening inside the vehicle, so the OAM module 3610 is notified in real time of the conversation and the driver's seat position. The OAM module 3610 also notices that the driver slightly moved his seat closer to his son and tilted it forward while they were talking.

[0247] During the journey:

[0248] 11. The autonomous vehicle begins to move toward the first handover position 3710. Since this first handover position is the most critical position, and the vehicle will require driver intervention, the HOH module 3630 begins to notify the driver in advance of the upcoming handover.

[0249] 12. The HOH module 3630 knows how long it may take for the driver to turn around and place his hands on the steering wheel.

[0250] 13. The HOH module 3630 also knows from the GOC database 3620 that experienced drivers typically take longer to become fully aware of the driving situation than younger drivers. Therefore, as an example, if the driver is a 50-year-old male, it might take him 15 to 20 seconds to become fully aware of the driving situation once he places his hands on the steering wheel. This additional time is therefore taken into account by the HOH module 3630 as the handover at position 3710 approaches.

[0251] 14. The HOH module 3630 also provides the EAO module 3635 with the driver's expected response time so that it can assess how to perform the handover. The driver responds to the handover request initiated by the vehicle, and he successfully guides the vehicle through an accident on the road.

[0252] 15. After leaving the accident site, the driver hands over to an autonomous vehicle when he receives an incoming call on his smartphone.

[0253] 16. EAO module 3635 begins to assess the handover at position 3710. It appears the driver responded 10 seconds later than expected by HOH module 3630. The timestamp on OAM module 3610 indicates that when the driver should have received control of the vehicle, he was busy handing a toy to his son, causing this unexpected delay.

[0254] 17. The anomaly is reported back to HOH module 3630 for future reference, in order to allow additional time for the planned handover.

[0255] 18. The driver's performance during the handover is also reported back to the POC module 3615 for internal updates.

[0256] 19. As the vehicle approaches handover 3720, OAM module 3610 confirms that the driver is still on the phone and appears to be issuing escalating emergency signals.

[0257] 20. The HOH module 3630 knows that the handover location 3720 can be avoided by following the alternative route 3715. This route will add an extra 2 minutes to the journey, but will allow the driver to continue his phone conversation without being interrupted.

[0258] 21. HOH module 3630 decides not to request handover at location 3720, and the vehicle continues to drive autonomously.

[0259] 22. HOH module 3630 is aware of road construction at handover location 3730, and although the handover at this location is not as critical as at location 3710, the travel time may be shorter with human intervention.

[0260] 23. The OAM module 3610 instructs the driver to stop making phone calls and to face forward and casually look at the traffic in front of the car.

[0261] 24. The HOH module 3630 determines that the driver may be able to take over easily and informs him of optional handovers to save travel time.

[0262] 25. After deciding that it was a good idea to save a few minutes by taking over, the driver agreed to take over, and the handover at location 3730 was successfully carried out.

[0263] 26. POC module 3615 was updated by EAO module 3635 after the handover, and since no anomaly was detected, HOH module 3630 did not receive a notification this time.

[0264] 27. For the remainder of the journey, the driver decided not to hand over the vehicle and drove to his destination in manual mode.

[0265] The example above is merely illustrative; more, fewer, or even different actions can be taken. Similarly, Figure 38 The example method is also exemplary, and more or fewer steps can be taken in this method.

[0266] It should also be noted that there may be situations where the autonomous vehicle has no choice but to hand over to a human driver (to achieve the original goals of the journey in terms of route, ETA, etc.), and the human driver is unable to take over. In such scenarios, the autonomous vehicle can choose from the following safer options: pull over and stop safely before the human driver can take over; move to the slower lane and reduce cruising speed, at the cost of increased travel time, depending on traffic and road conditions; calculate an alternative route that allows the vehicle to continue without handing over (this route may be longer and / or slower); or calculate an alternative route that does not allow the vehicle to continue to the final destination without handing over, but allows the vehicle to stop safely before the human driver is ready to take over. These solutions are merely exemplary, and there may be solutions that require mandatory handover of the vehicle.

[0267] Figures 39-40 This is a block diagram of an exemplary computer architecture that can be used according to the embodiments disclosed herein. Other computer architecture designs known in the art for processors and computing systems may also be used. Generally, suitable computer architectures for the embodiments disclosed herein may include, but are not limited to, those shown below. Figures 7-13 The configuration shown in the diagram. (39-40)

[0268] Figure 39 This is an example illustration of a processor according to an embodiment. Processor 3900 is an example of a type of hardware device that can be used in conjunction with the implementation described above. Processor 3900 can be any type of processor, such as a microprocessor, embedded processor, digital signal processor (DSP), network processor, multi-core processor, single-core processor, or other device for executing code. Although in Figure 39 The middle diagram shows only one processor 3900, but the processing element may alternatively include more than one. Figure 39 The processor 3900 is illustrated in the figure. The processor 3900 may be a single-threaded core, or in at least one embodiment, the processor 3900 may be multi-threaded, as each of its cores may include more than one hardware thread context (or "logical processor").

[0269] Figure 39Also illustrated is a memory 3902 coupled to a processor 3900 according to an embodiment. The memory 3902 can be any of a wide variety of memories (including layers of a memory hierarchy) known to those skilled in the art or otherwise available. Such memory elements can include, but are not limited to, random access memory (RAM), read-only memory (ROM), logic blocks of a field-programmable gate array (FPGA), erasable programmable read-only memory (EPROM), and electrically erasable programmable ROM (EEPROM).

[0270] Processor 3900 can execute any type of instructions associated with the algorithms, procedures, or operations described in detail herein. In general, processor 3900 can transform elements or artifacts (e.g., data) from one state or thing to another.

[0271] Code 3904 (which may be one or more instructions for execution by processor 3900) may be stored in memory 3902, or in software, hardware, firmware, or any suitable combination thereof, or, where appropriate and based on specific needs, in any other internal or external component, device, element, or object. In one example, processor 3900 may follow a program sequence of instructions indicated by code 3904. Each instruction enters front-end logic 3906 and is processed by one or more decoders 3908. The decoder may generate micro-operations, such as fixed-width micro-operations in a predefined format, as its output, or may generate other instructions, micro-instructions, or control signals that reflect the original code instructions. Front-end logic 3906 also includes register renaming logic 3910 and scheduling logic 3912, which generally allocates resources and queues operations corresponding to instructions available for execution.

[0272] Processor 3900 may also include execution logic 3914 having a set of execution units 3916a, 3916b, 3916n, etc. Some embodiments may include several execution units dedicated to a specific function or set of functions. Other embodiments may include only one execution unit or a single execution unit capable of performing a specific function. Execution logic 3914 performs operations specified by code instructions.

[0273] After the execution of the operation specified by the code instructions is completed, the back-end logic 3918 can decommission the instructions of code 3904. In one embodiment, processor 3900 allows out-of-order execution but requires ordered decommissioning of instructions. Decommissioning logic 3920 can take various known forms (e.g., reorder buffers, etc.). In this way, processor 3900 is translated during the execution of code 3904, at least for the output generated by the decoder, hardware registers, tables utilized by register renaming logic 3900, and any registers (not shown) modified by execution logic 3914.

[0274] Although not in Figure 39 As shown, the processing element may include other elements located on the chip along with the processor 3900. For example, the processing element may include memory control logic along with the processor 3900. The processing element may include I / O control logic and / or may include I / O control logic integrated with the memory control logic. The processing element may also include one or more caches. In some embodiments, non-volatile memory (such as flash memory or fuses) may also be included on the chip along with the processor 3900.

[0275] Figure 40 The figure illustrates a computing system 4000 arranged in a point-to-point (PtP) configuration according to an embodiment. Specifically, Figure 40 A system in which the processor, memory, and input / output devices are interconnected via several point-to-point interfaces is illustrated. In general, one or more computing systems described herein can be configured in the same or similar manner as computing system 3900.

[0276] Processors 4070 and 4080 may also each include integrated memory controller logic (MC) 4072 and 4082 for communication with memory elements 4032 and 4034. In alternative embodiments, memory controller logic 4072 and 4082 may be discrete logic separate from processors 4070 and 4080. Memory elements 4032 and / or 4034 may store various data used by processors 4070 and 4080 to implement the operations and functions outlined herein.

[0277] Processors 4070 and 4080 can be any type of processor, such as those discussed in conjunction with other figures herein. Processors 4070 and 4080 can exchange data via point-to-point interfaces 4050, respectively, using point-to-point (PtP) interface circuits 4078 and 4088. Processors 4070 and 4080 can each exchange data with chipset 4090 via respective point-to-point interfaces 4052 and 4054, using point-to-point interface circuits 4076, 4086, 4094, and 4098. Chipset 4090 can also exchange data with coprocessor 4038 via interface 4039, which may be a PtP interface circuit, such as a high-performance graphics circuit, a machine learning accelerator, or another coprocessor 4038. In an alternative embodiment, Figure 40 Any or all of the PtP links shown in the diagram can be implemented as a multi-branch bus instead of a PtP link.

[0278] Chipset 4090 can communicate with bus 4020 via interface circuitry 4096. Bus 4020 may have one or more devices communicating via it, such as bus bridge 4018 and I / O devices 4016. Bus bridge 4018 can communicate with other devices via bus 4010, such as user interfaces 4012 (e.g., keyboard, mouse, touchscreen, or other input devices), communication devices 4026 (e.g., modems, network interface devices, or other types of communication devices that can communicate via computer network 4060), audio I / O devices 4014, and / or data storage devices 4028. Data storage device 4028 may store code 4030, which can be executed by processors 4070 and / or 4080. In alternative embodiments, any part of the bus architecture may be implemented using one or more PtP links.

[0279] Figure 40 The computer systems depicted herein are schematic illustrations of embodiments of computing systems that can be used to implement the various embodiments discussed herein. It will be understood that... Figure 40 The components of the system described herein can be combined in a system-on-a-chip (SoC) architecture or in any other suitable configuration that enables the functionality and features of the examples and implementations provided herein.

[0280] While some of the systems and solutions described and illustrated herein have been described as comprising or associated with multiple elements, not all elements explicitly illustrated or described may be used in every alternative implementation of this disclosure. Furthermore, one or more of the elements described herein may be located outside the system, while in other instances, certain elements may be included within one or more other described elements and other elements not described in the illustrated implementations, or certain elements may be part of one or more other described elements and other elements not described in the illustrated implementations. In addition, certain elements may be combined with other components and may be used for alternative or additional purposes beyond those described herein.

[0281] Furthermore, it should be understood that the examples presented above are non-limiting examples provided for the purpose of illustrating certain principles and features only, and do not necessarily limit or constrain possible embodiments of the concepts described herein. For example, various different embodiments can be implemented using various combinations of the features and components described herein, including combinations implemented through various implementations of the components described herein. Other implementations, features, and details should be appreciated from the content of this specification.

[0282] Although this disclosure has been described in terms of certain implementations and generally associated methods, modifications and substitutions to these implementations and methods will be apparent to those skilled in the art. For example, the actions described herein may be performed in a different order than those described, and the desired result may still be obtained. As an example, the processes depicted in the appended figures do not necessarily require the specific order shown, or the sequential implementation of the desired result. In some implementations, multitasking and parallel processing may be advantageous. Additionally, other user interface layouts and functionalities may be supported. Other variations fall within the scope of the appended claims.

[0283] While this specification contains numerous specific implementation details, these details should not be construed as limiting the scope of any invention or claim, but rather as descriptions of features specific to particular embodiments of a particular invention. Certain features described in the context of individual embodiments within this specification may also be implemented in a single embodiment in combination. Conversely, various features described in the context of a single embodiment may also be implemented individually in multiple embodiments, or in any suitable sub-combination. Furthermore, while multiple features may be described above as functioning in certain combinations and even initially claimed in this way, in some cases, one or more features from a claimed combination may be removed from the combination, and the claimed combination may be for sub-combinations or variations thereof.

[0284] Similarly, although multiple operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring such operations to be performed in the specific order or sequence shown, or requiring all shown operations to be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of the various system components in the embodiments described above should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product, or packaged into nearly a number of software products.

[0285] One or more computing systems may be provided, including in-vehicle computing systems (e.g., for implementing at least a portion of an autonomous driving stack and enabling autonomous driving functions of a vehicle), roadside computing systems (e.g., separate from the vehicle; roadside computing systems implemented in dedicated roadside cabinets, on traffic signs, on traffic lights or lampposts, etc.), one or more computing systems implementing cloud-based or fog-based systems supporting autonomous driving environments, or computing systems located remotely from autonomous driving environments. These computing systems may include logic that performs or implements one or more of the following examples (or portions thereof) using one or a combination of: one or more data processing devices (e.g., central processing units, graphics processing units, tensor processing units, ASICs, FPGAs, etc.), accelerator hardware, other hardware circuitry, firmware, and / or software. For example, in various embodiments, the operation of the example methods below can be performed using any suitable logic, such as the computing system of a vehicle (e.g., 105) or its components (e.g., processor 202, accelerometer 204, communication module 212, user display 288, memory 206, IX architecture 208, driving control device 220, sensor 225, user interface 230, onboard processing system 210, machine learning model 256, other components or sub-components of any of these components), roadside computing device 140, fog or cloud-based computing system 150, drone 180, and access point 145, sensor (e.g., 165), memory 3602, processor core 3600, system 3700, other suitable computing systems or devices, sub-components of any of these components, or other suitable logic. In various embodiments, one or more specific operations of the example methods below may be performed by a specific component or system, while one or more other operations of the example methods may be performed by another component or system. In other embodiments, the operations of the example methods may each be performed by the same component or system.

[0286] Example 1 includes an apparatus comprising: at least one interface for receiving sensor data from a plurality of sensors of a vehicle; and one or more processors for: autonomously controlling the driving of the vehicle based on path planning according to the sensor data; determining that autonomous control of the vehicle should cease; sending a handover request to a telecomputing system so that the telecomputing system remotely controls the driving of the vehicle; receiving driving instruction data from the telecomputing system; and controlling the driving of the vehicle based on instructions included in the driving instruction data.

[0287] Example 2 includes the device of Example 1, wherein driving instruction data is generated based on input from a human user at a remote computing system.

[0288] Example 3 includes a device of any one of Examples 1-2, wherein one or more processors are configured to: detect a sidewalk parking event, wherein a vehicle stops driving in connection with and for the sidewalk parking event, wherein a handover request is sent in response to the sidewalk parking event.

[0289] Example 4 includes a device from any of Examples 1-2, wherein determining that autonomous control of a vehicle should cease includes using a specific machine learning model to predict conditions on the upcoming section that would hinder autonomous driving of the upcoming section.

[0290] Example 5 includes a device of any of Examples 1-4, wherein one or more processors are used to: determine, based on the detection of one or more damaged sensors of the vehicle, that autonomous control of the vehicle should be stopped.

[0291] Example 6 includes a device of any of Examples 1-5, wherein one or more processors are configured to: determine that no qualified passenger is present in the vehicle, wherein a handover request is sent at least in part based on the determination that no qualified passenger is present.

[0292] Example 7 includes a device of any of Examples 1-6, with one or more processors configured to: transmit sensor data to a remote computing system to present a dynamic representation of the vehicle’s surroundings to a human user of the remote computing system.

[0293] Example 8 includes the device of Example 7, wherein the sensor data includes video data.

[0294] Example 9 includes a device of any of Examples 1-8, with one or more processors for: sending an alert to passengers of a vehicle to indicate that control of the vehicle has been transferred to a remote valet parking service.

[0295] Example 10 includes a device of any of Examples 1-9, wherein one or more processors are used to: detect changes in conditions along a path planning route; and restore driving control of the vehicle from a remote computing system to the vehicle's autonomous driving logic.

[0296] Example 11 includes a computer-readable medium for storing instructions, wherein when executed by a machine, these instructions cause the machine to perform: autonomously controlling the driving of a vehicle based on path planning generated from a set of sensors of the vehicle; determining that autonomous control of the vehicle should cease; sending a handover request to a remote computing system so that the remote computing system can remotely control the driving of the vehicle; receiving driving instruction data from the remote computing system; and controlling the driving of the vehicle based on the instructions included in the driving instruction data.

[0297] Example 12 includes the medium of Example 11, wherein driving instruction data is generated based on input from a human user at a remote computing system.

[0298] Example 13 includes a medium of any of Examples 11-12, and the instructions, when executed by a machine, cause the machine to perform: detect a sidewalk event, wherein a vehicle is used to pull over and stops driving in association with the sidewalk event, wherein a handover request is sent in response to the sidewalk event.

[0299] Example 14 includes the medium of any of Examples 11-12, wherein determining that autonomous control of a vehicle should cease includes using a specific machine learning model to predict conditions on the upcoming section of the route planning that would make autonomous driving of the upcoming section difficult.

[0300] Example 15 includes a medium from any of Examples 11-14, wherein the determination is based on the detection of one or more damaged sensors on the vehicle to determine that autonomous control of the vehicle should be stopped.

[0301] Example 16 includes a medium of any of Examples 11-15, and the instructions, when executed by a machine, cause the machine to perform: determine that there is no qualified passenger present in the vehicle, wherein a handover request is sent based at least in part on the determination that there is no qualified passenger present.

[0302] Example 17 includes a medium of any of Examples 11-16, and the instructions, when executed by a machine, cause the machine to perform: send sensor data to a remote computing system to present a dynamic representation of the vehicle’s surroundings to a human user of the remote computing system.

[0303] Example 18 includes the medium of Example 17, wherein the sensor data includes video data.

[0304] Example 19 includes a medium of any of Examples 11-18, and the instructions, when executed by a machine, cause the machine to perform: issue an alert to the passengers of the vehicle to indicate that control of the vehicle has been transferred to a remote valet parking service.

[0305] Example 20 includes the medium of any of Examples 11-19, and the instructions, when executed by the machine, cause the machine to perform: detect changes in the conditions along the path planning; and restore driving control of the vehicle from the remote computing system to the autonomous driving logic of the vehicle.

[0306] Example 21 includes a system comprising: means for autonomously controlling the driving of a vehicle based on path planning generated from a set of sensors of the vehicle; means for determining that autonomous control of the vehicle should cease; means for sending a handover request to a telecomputing system so that the telecomputing system can remotely control the driving of the vehicle; means for receiving driving instruction data from the telecomputing system; and means for controlling the driving of the vehicle based on instructions included in the driving instruction data.

[0307] Example 22 includes the system of Example 21, in which driving instruction data is generated based on input from a human user at a remote computing system.

[0308] Example 23 includes a system comprising any one of Examples 21-22, further comprising: means for detecting a parking event, wherein a vehicle stops driving for parking at the side of the road and in association with the parking event, wherein a handover request is sent in response to the parking event.

[0309] Example 24 includes the system of Example 21, wherein determining that autonomous control of a vehicle should cease includes using a specific machine learning model to predict conditions on the upcoming section of the route planning that would cause difficulties for autonomous driving during the upcoming section.

[0310] Example 25 includes a vehicle comprising: a plurality of sensors for generating sensor data; a control system for physically controlling the movement of the vehicle; and processing circuitry for: autonomously controlling the driving of the vehicle based on path planning according to the sensor data by communicating with the control system; determining that autonomous control of the vehicle should cease; sending a handover request to a telecomputing system so that the telecomputing system can remotely control the driving of the vehicle; receiving driving instruction data from the telecomputing system; and controlling the driving of the vehicle based on instructions included in the driving instruction data by communicating with the control system.

[0311] Example 26 includes a method comprising: providing a user interface for a human user at a computing terminal device; receiving a handover request from a vehicle configured for autonomous driving; receiving sensor data from a remote sensor device describing the environment surrounding the vehicle; presenting a representation of the environment on the user interface based on the sensor data; receiving user input at the computing terminal device in response to the representation, wherein the user input includes input for guiding the driving of the vehicle within the environment; and sending instruction data to the vehicle corresponding to the user input to remotely drive the vehicle according to the user input.

[0312] Example 27 includes the method of Example 26, wherein the handover request identifies the location of the vehicle.

[0313] Example 28 includes the method of Example 27, further comprising: determining a sensor device corresponding to a location, wherein the sensor device is external to the vehicle; and accessing supplementary sensor data from the sensor device, wherein the representation is presented based at least in part on the supplementary sensor data.

[0314] Example 29 includes a method of any of Examples 26-28, wherein the sensor device includes a sensor device on a vehicle.

[0315] Example 30 includes a method of any one of Examples 26-29, wherein the sensor device includes a sensor device separate from the vehicle.

[0316] Example 31 includes the method of any one of Examples 26-30, further comprising: receiving a request from a vehicle to return driving control of the vehicle to the vehicle; sending an acknowledgment of the return of control to the vehicle; and ceasing the transmission of instruction data to the vehicle.

[0317] Example 32 includes the method of any one of Examples 26-30, further comprising: generating report data describing the environment and performance of the vehicle based on user input during control of the vehicle by a remote valet parking service; and sending the report data to a cloud-based system.

[0318] Example 33 is a system that includes means for performing the methods of any one of Examples 26-32.

[0319] Example 34 may include the system of Example 33, wherein the apparatus includes a computer-readable medium for storing instructions that, when executed by a machine, cause the machine to perform at least a portion of the method of any one of Examples 26-32.

[0320] Example 35 includes a method comprising: generating sensor data from a set of sensors on a vehicle; determining a path plan for the vehicle; autonomously controlling the driving of the vehicle based on the path plan, which is based on one or more machine learning models and sensor data; identifying conditions on an upcoming portion of the path plan; determining, based on the conditions, an opportunity to hand over driving control of the vehicle to a remote valet parking service; sending a handover request to a remote computing system, which provides the remote valet parking service, based on the opportunity; receiving driving instruction data from the remote computing system; and autonomously driving the vehicle in response to instructions contained in the instruction data.

[0321] Example 36 includes the method of Example 35, further including sending report data identifying the handover and the status corresponding to the handover to another computing system.

[0322] Example 37 includes the method of Example 36, in which report data is sent to a cloud-based application.

[0323] Example 38 includes a method from any of Examples 36-37, wherein report data is sent to a roadside unit.

[0324] Example 39 includes a method from any of Examples 35-38, wherein a condition is identified from data received from another computing system.

[0325] Example 40 includes the method of Example 39, wherein the condition is identified by applying a machine learning model, and data from another system is provided as input to the machine learning model.

[0326] Example 41 includes the method of Example 40, wherein the machine learning model is trained based on data from other instances of reports handed over to the remote valet parking service or parking incidents.

[0327] Example 42 includes a method from any of Examples 35-41, wherein a handover request is sent to avoid a pullover event.

[0328] Example 43 includes a method from any of Examples 35-42, wherein the chances of autonomous driving capability of the vehicle will correspond to a prediction of poor performance in the condition.

[0329] Example 44 includes the method of any of Examples 35-43, wherein the chance is determined at least in part based on information included in the sensor data.

[0330] Example 45 includes the method of any one of Examples 35-44, further comprising: accessing additional data; predicting improvements in the condition of the path planning following another part of the upcoming path based on the additional data; sending request data to a remote computing system to request a return of control to the vehicle based on the predicted improvements in the condition; and restoring autonomous control of driving the vehicle.

[0331] Example 46 includes a method from any of Examples 35-45, wherein determining the opportunity to transfer control includes detecting a pull-over event.

[0332] Example 47 includes the method of Example 46, further comprising: determining a condition based on sensor data associated with a parking incident; and uploading data describing the condition to a remote computing system.

[0333] Example 48 is a system that includes means for performing the methods of any one of Examples 35-47.

[0334] Example 49 may include the system of Example 48, wherein the means includes a computer-readable medium for storing instructions that, when executed by a machine, cause the machine to perform at least a portion of the method of any one of Examples 35-47.

[0335] Example 50 is a method comprising: generating a first set of one or more control signals in response to human input to a vehicle; identifying acceptable acceleration in response to determining that the first set of one or more control signals would result in unacceptable acceleration; converting the acceptable acceleration into a second set of one or more control signals; and providing the second set of one or more control signals to a vehicle actuation system in place of the first set of one or more control signals.

[0336] Example 51 includes the method of Example 50, further including: receiving a range of acceptable acceleration values; and identifying acceptable acceleration from the range of acceptable acceleration values.

[0337] Example 52 includes the method of Example 51, wherein the range of acceptable acceleration values ​​is determined based on an accident avoidance mathematical model.

[0338] Example 53 includes the method of any of Examples 51-52, wherein the range of acceptable acceleration values ​​is determined based on a responsibility-sensitive safety model.

[0339] Example 54 includes a method of any of Examples 50-53, wherein determining that one or more control signals will result in unacceptable acceleration includes using a machine learning model to convert one or more control signals into expected acceleration.

[0340] Example 55 includes a method of any one of Examples 50-54, wherein converting an acceptable acceleration into a second set of one or more control signals includes: converting the acceptable acceleration based on a context associated with the vehicle, wherein the context is determined based on inputs received via one or more sensors of the vehicle.

[0341] Example 56 includes the method of Example 55, wherein input received via one or more sensors of the vehicle indicates one or more of the following: road conditions, weather conditions, tire conditions, or road layout.

[0342] Example 57 includes a method of any one of Examples 50-56, wherein converting an acceptable acceleration into a second set of one or more control signals includes converting the acceptable acceleration based on the vehicle's weight.

[0343] Example 58 includes a method of any of Examples 50-57, wherein identifying an acceptable acceleration includes selecting an acceptable acceleration from a range of acceptable accelerations based on strategy information provided by the driver of the vehicle.

[0344] Example 59 includes the method of any one of Examples 50-58, further comprising: generating a third set of one or more control signals in response to human input to the vehicle; and providing the third set of one or more control signals to the vehicle actuation system without alteration in response to determining that the third set of one or more control signals will result in acceptable acceleration.

[0345] Example 60 includes an apparatus comprising a memory and processing circuitry coupled to the memory, the processing circuitry being used to perform one or more of the methods of Examples 50-59.

[0346] Example 61 includes a system comprising means for performing one or more of the methods of Examples 50-59.

[0347] Example 62 includes at least one machine-readable medium, including instructions, wherein when executed, these instructions implement a device or a method as described in any of Examples 50-59.

[0348] Example 63 includes a method comprising: determining a signal quality metric by a vehicle's computing system based on sensor data and the context of the sensor data; determining, based on the signal quality metric, the likelihood of a safety associated with a transfer of control of the vehicle; and preventing or initiating a transfer of control of the vehicle based on the likelihood of safety.

[0349] Example 64 includes the method of Example 63, and further includes using a machine learning model to determine the context of the sensor data based on the sensor data.

[0350] Example 65 includes the method of any of Examples 63-64, and further includes using a machine learning model to determine the possibility of security based on a signal quality metric.

[0351] Example 66 includes the method of any of Examples 63-65, and further includes using a machine learning model to determine a signal quality metric based on sensor data and the context of the sensor data.

[0352] Example 67 includes the method of any one of Examples 63-66, further including periodically determining the likelihood of safety associated with the transfer of control of the vehicle when the vehicle is autonomously controlled.

[0353] Example 68 includes the method of any of Examples 63-67, further including determining the likelihood of safety associated with the transfer of control of the vehicle in response to a request from a human driver for the transfer of control of the vehicle.

[0354] Example 69 includes the method of any of Examples 63-68, further including determining the possibility of security associated with the transfer of control of the vehicle in response to a high-resolution map of the area in which the vehicle is unavailable.

[0355] Example 70 includes a method of any of Examples 63-69, wherein the signal quality metric at least partially indicates the signal-to-noise ratio of the sensor data.

[0356] Example 71 includes a method of any of Examples 63-70, wherein the signal quality metric at least partially indicates the resolution of the sensor data.

[0357] Example 72 includes an apparatus comprising a memory and processing circuitry coupled to the memory, the processing circuitry being used to perform one or more of the methods of Examples 63-71.

[0358] Example 73 includes a system comprising means for performing one or more of the methods of Examples 63-71.

[0359] Example 74 includes at least one machine-readable medium, including instructions, wherein when executed, these instructions implement a device or a method as described in any of Examples 63-71.

[0360] Example 75 includes a method comprising: collecting sensor data from at least one sensor located inside a vehicle; analyzing the sensor data to determine the physical condition of a person inside the vehicle; and generating a handover decision based at least in part on the physical condition of the person, the handover decision indicating whether the person is expected to be able to operate the vehicle safely.

[0361] Example 76 includes the method of Example 75, further including: identifying historical driving data of persons inside the vehicle; and further generating a handover decision based on the historical driving data of the persons.

[0362] Example 77 includes the method of any one of Examples 75-76, further comprising: analyzing sensor data to determine a context indicating conditions outside the vehicle; and further generating a handover decision based on the context.

[0363] Example 78 includes the method of any one of Examples 75-77, wherein the physical state of occupants inside the vehicle is based at least in part on sensor data including image data of the occupants inside the vehicle.

[0364] Example 79 includes the method of any one of Examples 75-78, wherein the physical state of occupants inside the vehicle is based at least in part on sensor data including audio data of the occupants inside the vehicle.

[0365] Example 80 includes the method of any one of Examples 75-79, wherein the physical condition of occupants inside the vehicle is based at least in part on sensor data including temperature data of the occupants inside the vehicle.

[0366] Example 81 includes the method of any one of Examples 75-80, wherein the physical state of occupants inside a vehicle is based at least in part on sensor data including pressure data from tactile sensors.

[0367] Example 82 includes a method of any of Examples 75-81, wherein the physical condition of a person inside a vehicle is based at least in part on data received from a health tracking device worn by the person.

[0368] Example 83 includes the method of any one of Examples 75-82, further comprising: determining a specific activity being performed by a person inside a vehicle based on sensor data; and wherein the physical state of the person inside the vehicle is at least in part based on the determined activity.

[0369] Example 84 includes the method of any one of Examples 75-83, further comprising: preprocessing audio data of sensor data to isolate sounds caused by occupants or one or more passengers inside the vehicle; and wherein the physical state of the occupants is based at least in part on the preprocessed audio data.

[0370] Example 85 includes a method of any one of Examples 75-84, wherein the sensor data includes one or more of the following: media being played in the vehicle; light levels inside the vehicle; the amount of interaction between a person and one or more dashboard controls; window aperture levels; the status of the cabin temperature control system; or the status of a person's telephone.

[0371] Example 86 includes a method from any of Examples 75-85, wherein the physical state of a person is determined by a machine learning algorithm that uses sensor data as input.

[0372] Example 87 includes the method of any of Examples 75-86, and further includes using machine learning algorithms to generate handover decisions.

[0373] Example 88 includes an apparatus comprising a memory and processing circuitry coupled to the memory, the processing circuitry being used to perform one or more of the methods of Examples 75-87.

[0374] Example 89 includes a system comprising means for performing one or more of the methods of Examples 75-87.

[0375] Example 90 includes at least one machine-readable medium, including instructions, wherein when executed, these instructions implement a device or a method as described in any of Examples 1-13.

[0376] Example 91 includes a method comprising: operating an autonomous vehicle in an autonomous driving mode by a controller of the autonomous vehicle; receiving a request from an entity other than the controller to take over control of the autonomous vehicle; prompting the requesting entity to provide credentials in response to receiving the request to take over control of the autonomous vehicle; receiving input in response to the prompt; and authenticating the requesting entity based on the received input to allow the request to take over control of the autonomous vehicle.

[0377] Example 92 includes the method of Example 91, wherein prompting the requesting entity to provide credentials includes prompting the requesting entity to provide biometrics for authentication.

[0378] Example 93 includes the method of Example 92, wherein biometrics includes one or more of the following: fingerprints, voice samples for speech recognition, and facial samples for face recognition.

[0379] Example 94 includes a method from any of Examples 91-93, wherein the requesting entity includes persons inside an autonomous vehicle.

[0380] Example 95 includes a method from any of Examples 91-93, wherein the requesting entity includes a person who is remote from autonomous transportation.

[0381] Example 96 includes a method of any of Examples 91-93, wherein the requesting entity includes one or more other autonomous vehicles located near the autonomous vehicle.

[0382] Example 97 includes an apparatus comprising a memory and processing circuitry coupled to the memory, the processing circuitry being used to perform one or more of the methods of Examples 91-96.

[0383] Example 98 includes a system comprising means for performing one or more of the methods of Examples 91-96.

[0384] Example 99 includes a product comprising one or more tangible computer-readable nontransitory storage media, including computer-executable instructions that, when executed by at least one computer processor, are operable to enable the at least one computer processor to perform the methods of Examples 91-96.

[0385] Example 100 includes a method comprising: operating an autonomous vehicle in a manual operation mode, wherein the autonomous vehicle is controlled based on human input; receiving sensor data from multiple sensors within the autonomous vehicle; detecting that the human input is unsafe based on analysis of the sensor data; and operating the autonomous vehicle in an autonomous operation mode in response to detecting the unsafe human input.

[0386] Example 101 includes the method of Example 100, wherein detecting that human input is unsafe includes one or more of the following: determining that the human providing the input is distracted, determining that the human providing the input is compromised, and determining that the human providing the input is unconscious.

[0387] Example 102 includes an apparatus comprising a memory and processing circuitry coupled to the memory, the processing circuitry being used to perform one or more of the methods of Examples 100-101.

[0388] Example 103 includes a system comprising means for performing one or more of the methods of Examples 100-101.

[0389] Example 104 includes a product comprising one or more tangible computer-readable nontransitory storage media, including computer-executable instructions that, when executed by at least one computer processor, are operable to enable the at least one computer processor to perform the methods of Examples 100-101.

[0390] Example 105 includes a method comprising: operating an autonomous vehicle in an autonomous operation mode by a control system of the autonomous vehicle based on sensor data obtained from a plurality of sensors coupled to the autonomous vehicle; detecting a takeover request made by a passenger of the autonomous vehicle by the control system of the autonomous vehicle; determining, based on the sensor data, whether the requested takeover is safe; and preventing the requested takeover in response to determining that the requested takeover is unsafe.

[0391] Example 106 includes the method of Example 105, and further includes modifying the autonomous operating mode in response to determining that the requested takeover is unsafe.

[0392] Example 107 includes the method of Example 106, further comprising: prompting a passenger to provide input in response to the determination; and receiving input from the passenger in response to the prompt; wherein the autonomous operation mode is modified based on the received input.

[0393] Example 108 includes the method of Example 105, wherein multiple sensors coupled to the autonomous vehicle include internal sensors inside the autonomous vehicle, and a determination is made based on sensor data received from the internal sensors as to whether the requested takeover is safe.

[0394] Example 109 includes the method of Example 108, wherein the internal sensors include one or more of a camera and a microphone.

[0395] Example 110 includes a method of any one of Examples 105-109, and further includes allowing the takeover request in response to determining that the requested takeover is normal.

[0396] Example 111 includes a method of any of Examples 105-109, and further includes blocking a takeover request in response to determining that the requested takeover is insecure.

[0397] Example 112 includes a method of any one of Examples 105-111, wherein a determination of whether a requested takeover is unsafe is performed during the sensing / perception phase of the autonomous driving pipeline.

[0398] Example 113 includes a method of any of Examples 105-112, wherein the requested takeover is prevented from being performed during the behavior / control phase of the autonomous driving pipeline.

[0399] Example 114 includes the method of Example 107, wherein the modification of the autonomous operating mode is performed during the planning phase of the autonomous driving pipeline.

[0400] Example 115 includes an apparatus that includes a memory and processing circuitry coupled to the memory, the processing circuitry being used to perform one or more of the methods of Examples 105-114.

[0401] Example 116 includes a system comprising means for performing one or more of the methods of Examples 105-114.

[0402] Example 117 includes a product comprising one or more tangible computer-readable nontransitory storage media, including computer-executable instructions that, when executed by at least one computer processor, are operable to enable the at least one computer processor to perform the methods of Examples 105-114.

[0403] Example 118 includes a method comprising: monitoring at least one subsystem of an autonomous vehicle by a monitoring system; and initiating a change in the autonomy level of the autonomous vehicle from a first autonomy level to a second autonomy level by the monitoring system based on the monitoring of the at least one subsystem.

[0404] Example 119 includes the method of Example 118, and further includes transmitting changes in the autonomy level of an autonomous vehicle to a remote monitoring system.

[0405] Example 120 includes the method of any of Examples 118-119, and further includes recording the history of autonomy level and sensor state over time.

[0406] Example 121 includes the method of any one of Examples 118-120, wherein at least one subsystem includes a sensor subsystem, and the change in the level of autonomy is based at least in part on the change in the sensor subsystem.

[0407] Example 122 includes one or more of the methods in Examples 118-121, wherein at least one subsystem includes a planning subsystem, and the change in the level of autonomy is based at least in part on a change to the planning subsystem.

[0408] Example 123 includes one or more of the methods in Examples 118-122, wherein at least one subsystem includes an execution subsystem, and changes to the level of autonomy are based at least in part on changes to the execution subsystem.

[0409] Example 124 includes one or more of the methods in Examples 118-123, wherein a monitoring system is used to monitor the functional guarantees of at least one subsystem.

[0410] Example 125 includes one or more of the methods in Examples 118-124, wherein a comprehensive cognitive supervision system monitors the quality assurance of at least one subsystem.

[0411] Example 126 includes an apparatus that includes a memory and processing circuitry coupled to the memory, the processing circuitry being used to perform one or more of the methods of Examples 118-125.

[0412] Example 127 includes a system comprising means for performing one or more of the methods of Examples 118-125.

[0413] Example 128 includes at least one machine-readable medium, including instructions, wherein when executed, these instructions implement a device or a method as described in any of Examples 118-125.

[0414] Example 129 includes a method comprising: determining a system malfunction of an autonomous vehicle; determining that the autonomy level of the autonomous vehicle can be reduced to a first level that does not require driver intervention; alerting the driver that the autonomy level will be reduced to the first level; and reducing the autonomy level to the first level.

[0415] Example 130 includes the method of Example 129, further comprising: determining that an additional system malfunction exists in the autonomous vehicle; determining that the autonomy level can be reduced to a second level; alerting the driver that the autonomy level will be reduced to a second level; and reducing the autonomy level to a second level.

[0416] Example 131 includes one or more of the methods in Examples 129-130, and further includes confirming the driver's participation.

[0417] Example 132 includes the method of Example 131, wherein confirming driver participation includes monitoring the driver.

[0418] Example 133 includes one or more of the methods in Examples 129-132, and further includes: determining that an additional system failure exists in the autonomous vehicle; determining that the autonomy of the vehicle must be deactivated; and attempting to hand over to the driver in response to determining that the autonomy of the vehicle must be deactivated.

[0419] Example 134 includes the method of Example 133, and further includes determining whether the handover was successful.

[0420] Example 135 includes the method of Example 134, and further includes: deactivating the autonomy of the vehicle if the handover is successful.

[0421] Example 136 includes the method of Example 134, and further includes activating the emergency system if the handover is unsuccessful.

[0422] Example 137 includes the method of Example 136, wherein an emergency system is used to safely stop an autonomous vehicle.

[0423] Example 138 includes a system comprising means for performing any one or more of Examples 129-137.

[0424] Example 139 includes the system of Example 138, wherein the apparatus includes at least one machine-readable medium including instructions, wherein the instructions, when executed, implement one or more of the methods of Examples 129-137.

[0425] Example 140 includes a method comprising: determining at least one handover point between an autonomous vehicle and a driver on a route; receiving information relating to driver characteristics; receiving information relating to the driver’s current state of attention; and determining expected driver behavior during each handover point at the at least one handover point.

[0426] Example 141 includes the method of Example 140, wherein information relating to driver characteristics includes general information.

[0427] Example 142 includes one or more of the methods in Examples 140-141, wherein information relating to driver characteristics includes driver-specific information.

[0428] Example 143 includes one or more of the methods in Examples 140-142, and further includes determining whether the driver is ready to make a handover.

[0429] Example 144 includes the method of Example 143, further including transferring control of the vehicle to the driver in response to determining that the driver is ready to make a handover.

[0430] Example 145 includes the method of Example 143, and further includes: calculating alternative handover options if the driver is not ready to hand over the work.

[0431] Example 146 includes the method of Example 145, wherein the alternative includes finding an alternative route.

[0432] Example 147 includes the method of Example 145, wherein the alternative includes stopping the vehicle.

[0433] Example 148 includes one or more of the methods in Examples 140-147, and further includes updating information related to the characteristics of the driver.

[0434] Example 149 includes a system comprising means for performing any one or more of Examples 140-148.

[0435] Example 150 includes the system of Example 149, wherein the apparatus includes at least one machine-readable medium including instructions, wherein the instructions, when executed, implement the method of any one or more of Examples 140-148.

[0436] Example 151 includes a system comprising: an occupant activity monitoring module; a personalized occupant capability database; a general occupant capability database; a handover prediction module; an execution assessment and optimization module; and a handover handling module.

[0437] Therefore, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. In some cases, the actions listed in the claims can be performed in a different order and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific order or sequence shown to achieve the desired result.

Claims

1. An apparatus comprising a processor configured to: collect sensor data from at least one sensor located inside a vehicle, the vehicle being controlled by a driving system of the vehicle; analyze the sensor data to determine a physical state of a person inside the vehicle; and generate a decision to transfer control back from the driving system of the vehicle to a driver of the vehicle based at least in part on the physical state of the person.

2. The apparatus of claim 1, the processor further configured to alert the driver that control of the vehicle is to change from the driving system to the driver based on the physical state of the driver.

3. The apparatus of claim 2, the processor further configured to activate an emergency system of the vehicle if the driver does not gain control after a time period following the alert to the driver.

4. The apparatus of claim 3, wherein, activating the emergency system of the vehicle comprises causing the vehicle to pull over.

5. The apparatus of any of claims 1-4, wherein, the decision indicates whether the person is expected to be able to safely operate the vehicle.

6. The apparatus of any of claims 1-5, wherein, the driving system is an autonomous driving system.

7. A computer-readable non-transitory storage medium comprising computer- executable instructions operable to, when executed by at least one computer processor, cause the at least one computer processor to implement operations of a method comprising: collecting sensor data from at least one sensor located inside a vehicle, the vehicle being controlled by a driving system of the vehicle; analyzing the sensor data to determine a physical state of a person inside the vehicle; and generating a decision to transfer control back from the driving system of the vehicle to a driver of the vehicle based at least in part on the physical state of the person. alerting the driver that control of the vehicle is to change from the driving system to the driver based on the physical state of the driver.

8. The computer-readable non-transitory storage medium of claim 7, the method further comprising: activating an emergency system of the vehicle if the driver does not gain control after a time period following the alert to the driver.

9. The computer-readable non-transitory storage medium of claim 8, the method further comprising: activating the emergency system of the vehicle comprises causing the vehicle to pull over.

10. The computer-readable non-transitory storage medium of claim 9, wherein, the decision indicates whether the person is expected to be able to safely operate the vehicle.

11. The computer-readable non-transitory storage medium of any of claims 7-10, wherein, the driving system is an autonomous driving system.

12. The computer-readable non-transitory storage medium of any of claims 7-11, wherein, 13. A vehicle comprising the apparatus of any of claims 1-6.

14. A control unit of a vehicle comprising the apparatus of any of claims 1-6. ​

Images