CONTROLLING FUNCTIONS AND OUTPUTS OF AUTONOMOUS VEHICLES BASED ON THE POSITION AND ATTENTION OF OCCUPANTS

Abstract

System for use with an autonomous vehicle (10), comprising: a non-temporary storage device (104), comprising: a context module (314) of the autonomous vehicle (10) which, when executed by a hardware-based processing unit (106), determines whether manual operation of the vehicle (10) is recommended or required during autonomous operation of the vehicle (10); and a driver module (312) of the autonomous vehicle (10) which, when executed by the hardware-based processing unit (106), determines, based on occupant position data received from an occupant position sensor, that a vehicle occupant is not ready to take manual control of the autonomous vehicle (10); wherein the occupant position sensor is part of a portable occupant device (31, 32, 33) which can be used both as an input source and for output communication.

Description

TECHNICAL AREA

[0001] The present invention relates generally to systems for controlling vehicle functions and in particular to systems, algorithms and processes for controlling functions and outputs of vehicles, including autonomous vehicles, based on an occupant position and / or occupant attention during operation of the vehicle. BACKGROUND

[0002] This section provides background information relating to the present disclosure, which may not necessarily be prior art.

[0003] Manufacturers are increasingly producing vehicles with higher levels of automated driving. Features such as adaptive cruise control and lateral positioning have become popular and are precursors to wider application in vehicles capable of fully autonomous driving.

[0004] While the availability of autonomous driving-capable vehicles is increasing, user familiarity with and comfort level with autonomous driving features may not necessarily keep pace. User comfort with automation is a crucial aspect for the overall acceptance of the technology and for the user's driving experience.

[0005] Drivers or other vehicle occupants using the autonomous functionality may not be familiar with various aspects of being a passenger in autonomous vehicles, which may decrease satisfaction with the driving experience.

[0006] Furthermore, drivers who use autonomous functionalities may not be used to taking back control of the vehicle, which can be stressful.

[0007] German patent application DE 10 2013 019 141 A1 discloses a driving mode change in a driver assistance system in which sensors record driver data that depends on the body posture of at least one part of the driver. If the recorded driver data meets a predetermined safety condition, the system can switch from an autonomous driving mode to another predetermined driving mode.

[0008] In the publication DE 10 2014 220 050 A1, a system and a method for assisting a driver of a motor vehicle are disclosed, in which a detection device detects a body posture of a driver located in the motor vehicle, an output device outputs information to the driver, and a control device determines at least one parameter that characterizes a change in the detected body posture and controls the motor vehicle accordingly.

[0009] German patent application DE 10 2017 103 973 A1 discloses an autonomous vehicle control system that takes driver attention into account. This system receives signals from multiple sources representing aspects of a vehicle, a driver, and the vehicle's environment. From these signals, an attention factor and a readiness factor are determined, and based on these, the system switches between different levels of autonomous control.

[0010] German patent application DE 10 2016 200 513 A1 discloses a method and a device for operating a motor vehicle in which a transition from a driving mode with at least partial automation of the vehicle control to a driving mode with manual vehicle control is achieved by determining at least one parameter characteristic of the driver's current status. A transition strategy for the change between the two driving modes is then adapted based on this parameter.

[0011] The publication DE 10 2014 224 765 A1 discloses a device and a method for driver activity detection, in which a camera records image data and a computing unit determines an area around the eyes of a driver from the image data of the camera and stores it in a storage unit, since the eyes allow many inferences.

[0012] In US patent 10,133,270 B2, an electronic control unit for switching vehicle control out of an autonomous driving mode is disclosed, wherein the electronic control unit determines that the autonomous driving mode is about to end, determines that a driver is not able to immediately assume control of the vehicle upon termination of the autonomous driving mode, and transfers control of the vehicle to a remote operator.

[0013] Document US 2017 / 0028987A1 discloses a vehicle control system with a device for acquiring information about vehicle occupants, in which a microcomputer controls the switching between autonomous and manual driving modes. The device detects the fitness to drive of the vehicle occupants. When switching from autonomous to manual driving mode, the microcomputer alerts one of the occupants whose fitness to drive has been determined.

[0014] German patent application DE 10 2015 223 248 A1 discloses a method for a driver assistance system for the automated control of a vehicle, in which a predicted end of an automated driving segment and a preliminary handover point are determined. Objects in the vehicle's environment are detected from sensor data, stored data, and received data, and action-relevant objects are determined from this data during a handover interval.

[0015] The publication DE 10 2016 219 795 A1 discloses a control method for an autonomous vehicle in which a takeover situation is detected during autonomous control of the vehicle and control functions are transferred to an occupant of the vehicle, whereby gestures of the occupant are recorded in the vehicle, control instructions are recognized from them and a sequence of control commands is created and executed.

[0016] In the publication DE 10 2016 206 126 A1, a method for monitoring or controlling a driving task handover in a self-driving vehicle is disclosed, in which driving task takeover information, representing an intended takeover of the driving task by an occupant of the vehicle, is read from an interface assigned to a vehicle seat of the vehicle, sensor information about an occupancy state of the vehicle seat is read, and the driving task takeover information and the sensor information are used to release or control the driving task handover. SUMMARY

[0017] In one aspect, the present disclosure relates to systems for use with an autonomous vehicle, which include a non-temporary storage device comprising a context module of an autonomous vehicle. This context module, when executed by a hardware-based processing unit, determines whether manual operation of the vehicle is recommended or necessary during autonomous operation. The storage device also includes a driver module of an autonomous vehicle. When executed, this driver module, based on occupant position data indicating the location of the vehicle occupant and / or occupant gaze data indicating the occupant's gaze, determines whether a vehicle occupant is ready to assume manual control of the autonomous vehicle.In response, the system can advise the user to assume a correct position in order to take over manual control of the vehicle. Furthermore, the system can correct the current autonomous driving behavior, for example, by slowing the vehicle down or steering it towards a parking position.

[0018] In another aspect, the present disclosure relates to a process for promoting the readiness of a driver in an autonomously driving transport vehicle. The process comprises: (i) the autonomous vehicle's passenger position sensor detecting the position of an occupant positioned in the autonomous vehicle, thereby providing occupant position data; and / or (ii) the autonomous vehicle's occupant gaze sensor detecting the occupant's gaze, thereby providing occupant gaze data.

[0019] The occupant position can encompass various aspects of the position, such as the location, pose, and orientation of the occupant in general, or of one or more of the occupant's body parts, such as their head or face.

[0020] In various embodiments, the process also includes a hardware-based processing unit, which executes a context module of autonomous driving, determining during autonomous operation of the vehicle that manual operation of the vehicle is recommended or necessary.

[0021] The process further includes the hardware-based processing unit, which executes a driver module of the autonomous vehicle, determining, based on occupant position data and / or occupant gaze data, that the occupant is not ready to take over manual control of the autonomous vehicle.

[0022] The process further includes the determination of an action by the hardware-based processing unit executing an activity module, and the initiation of the action in response to the finding that the occupant is not ready to take manual control of the autonomous vehicle.

[0023] In various embodiments, the action includes communicating with the occupant to motivate the occupant to assume a position in order to take over manual control of the autonomous vehicle.

[0024] The action may include transmitting an alarm via a human-machine interface (HMI) of the autonomous vehicle for the occupant to receive, in order to motivate the occupant to take up a position and / or concentrate their attention in order to take over manual control of the autonomous vehicle.

[0025] The action may involve altering the vehicle's autonomous driving mode, for example by adjusting an autonomous driving actuator. In various embodiments, the autonomous driving actuator comprises at least one actuation component selected from a group consisting of a steering component, a braking component, and a throttle component.

[0026] The measure of determining that manual control of the vehicle is recommended or necessary may include determining that manual operation is recommended or necessary for safety reasons; or determining that manual operation is recommended or necessary based on a pre-defined setting or user preference.

[0027] In various embodiments, detecting the position of the occupant positioned in the autonomous vehicle includes detecting the respective positions of each of several occupants in the vehicle, thereby providing occupant position data; and determining that the occupant is not ready to assume manual control of the autonomous vehicle includes determining that none of the occupants is ready to assume control of the vehicle.

[0028] In some implementations, the process involves determining which occupant is in the best position to take over manual control of the autonomous vehicle.

[0029] In various embodiments, the process involves determining the occupant's gaze based on occupant gaze data and vehicle position data, and determining that the occupant is not ready to assume manual control of the autonomous vehicle based on the occupant gaze.

[0030] While occupant gaze is detected in various embodiments using one or more vehicle sensors, another aspect of the technology involves a process for monitoring the attention of an occupant of the autonomous vehicle who is wearing smart glasses or holding a mobile communication device—such as a tablet computer—and for selectively interacting with them. This process involves the autonomous vehicle receiving gaze data from the occupant's glasses, which contain an occupant gaze sensor, indicating the occupant's gaze while driving autonomously.

[0031] The process involves a hardware-based processing unit determining the orientation of the occupant's eyes in a reference frame when autonomous vehicle functions are executed, thereby providing orientation data of the occupant of the autonomous vehicle.

[0032] The process also includes: the hardware-based processing unit determining an action when executing an activity module based on the autonomous vehicle's gaze data and occupant orientation data; and initiating the execution of the action.

[0033] In various embodiments, the action comprises one or more of: (a) that communication is provided via the occupant's glasses to alert the occupant to an object in the occupant's field of vision in the autonomous vehicle; (b) that communication is provided via a human-machine interface (HMI) of the vehicle to alert the occupant to an object in the occupant's field of vision in the autonomous vehicle; and (c) that communication is provided to alert the occupant to an object in the occupant's field of vision in the autonomous vehicle.

[0034] The object can comprise any of: (I) an occupant destination indicated by an occupant's travel route; and (II) an advertisement outside the vehicle. In one embodiment, the process includes initiating an advertisement subprocess in response to the detection that the occupant is currently viewing the advertisement, which credits a corresponding account.

[0035] In various implementations, the process involves the hardware-based processing unit, when executing an activity module, determining, based on the autonomous vehicle's gaze data and occupant orientation data, that the occupant has viewed an advertisement. The action may then involve initiating an advertisement subprocess, which awards a credit corresponding to the occupant currently viewing the advertisement.

[0036] The disclosure also includes systems containing any of the described components - e.g., the hardware-based processing unit, modules, etc. - to perform any of the operations described above.

[0037] And the disclosure includes computer-readable storage devices, - e.g. custom modules for execution by a hardware-based processing unit - designed to perform any of the operations described above.

[0038] Other aspects of the present technology will be partly revealed and partly disclosed below. DESCRIPTION OF THE DRAWINGS Fig. Figure 1 schematically illustrates an exemplary transport vehicle with local and remote personal computing devices in accordance with embodiments of the present technology. Fig. Figure 2 schematically illustrates more details of the exemplary vehicle computer from Fig. 1 in communication with the local and remote computing devices. Fig. Figure 3 shows a different view of the vehicle, highlighting exemplary storage components. Fig. 4 shows interactions between the different components of Fig. 3, which include external systems.

[0039] The figures are not necessarily to scale and some features may be emphasized or minimized to show details of specific components.

[0040] The figures show exemplary embodiments and therefore the technology is not limited to the examples shown. DETAILED DESCRIPTION

[0041] As requested, detailed embodiments of the present disclosure are disclosed here. The disclosed embodiments are merely examples that can be implemented in various and alternative forms and combinations thereof. The terms "for example," "exemplary," and similar terms, as used herein, extensively denote embodiments that serve as illustrations, examples, models, or patterns.

[0042] In some cases, well-known components, systems, materials, or processes have not been described in detail to avoid obscuring the present disclosure. Therefore, specific structural and functional details disclosed here should not be interpreted as limitations, but merely as a basis for the claims and as a representative basis for informing the person skilled in the art about how to use the present disclosure. I. Technological Introduction

[0043] The present disclosure describes, by various embodiments, systems for controlling vehicle functions and in particular systems, algorithms and processes for controlling functions and outputs of autonomous vehicles based on an occupant position and / or occupant attention during autonomous driving.

[0044] Functions can include providing communications for the occupant, such as an alarm or a message instructing the occupant to take over control of the vehicle. Another example of communication would be to inform the occupant that the vehicle is currently transitioning from autonomous to manual driving.

[0045] Another example function involves directing an occupant's gaze to a target object. Determining that the gaze has been directed somewhere is performed by one or more vehicle sensors or using an occupant device, such as smart glasses or a mobile communication device like a tablet or phone.

[0046] While selected examples of the present technology describe transport vehicles or modes of travel, and specifically motor vehicles, the technology is not limited to this focus. The concepts can be extended to a wide variety of systems and devices, such as other transport or movement vehicles, including aircraft, watercraft, trucks, buses, and the like.

[0047] Although selected examples of the present technology describe autonomous vehicles, the technology is not limited to use in autonomous vehicles (fully or partially autonomous) or to periods during which a vehicle capable of autonomous driving is actually being driven autonomously. For example, references made herein to occupant characteristics and to communications made available for reception by an occupant should be understood as revealing analogous implementations with respect to a vehicle driver during manual vehicle operation. During fully autonomous driving, the "driver" is considered an occupant or passenger, and the terms occupant and passenger may sometimes be used interchangeably in the claims and elsewhere herein unless a distinction is explicitly stated. II. Carrier vehicle - Fig. 1

[0048] Referring now to the characters, and in particular to the first character, it shows Fig. 1 an exemplary support structure or support device 10 in the form of a vehicle.

[0049] The vehicle 10 contains a hardware-based controller or controller system 20. The hardware-based controller system 20 contains a communication subsystem 30 for communication with mobile or local computing devices 34 and / or with external networks 40.

[0050] Through external networks 40, such as the Internet, a local network, a mobile network or a satellite network, through vehicle-to-vehicle, pedestrian-to-vehicle or other infrastructure communications, etc., the vehicle 10 can reach mobile or local systems 34 or remote systems 50, such as remote servers.

[0051] Examples of mobile or local devices 34 include a passenger's smartphone 31, a passenger's body-worn device 32, and a tablet computer 33, and are not limited to these examples. Examples of body-worn devices 32 include smartwatches, glasses, as shown, and smart jewelry, such as earrings, necklaces, bows, etc.

[0052] Another example of a mobile or local device is an on-board device (OBD), such as a wheel speed sensor, brake pressure sensor, accelerometer, brake disc wear sensor, throttle position sensor, steering angle sensor, revolutions per minute (RPM) indicator, brake force sensors, or any other sensor for a vehicle condition or dynamics, with which the vehicle is retrofitted after manufacture. The OBDs may include or be part of the sensor subsystem referred to below by reference numeral 60.

[0053] The sensors 60 comprise sensors 601, 602 focused on the interior of the vehicle, such as microphones and cameras, designed to detect the presence of persons, the activities of persons, or other activities or properties within the passenger compartment. This subset of sensors is described in more detail below.

[0054] The vehicle controller system 20, which in the considered embodiments includes one or more microcontrollers, can communicate with OBDs via a controller area network (CAN). The message-based CAN protocol is typically designed for electrical multiplex wiring in motor vehicles, and the CAN infrastructure may include a CAN bus. The OBDs may also be referred to as components or products with a vehicle CAN interface (VCI components or products), and the signals transmitted via CAN may be referred to as CAN signals. Communications between the OBDs and the primary controller or microcontroller 20 are performed in other embodiments using similar or different message-based protocols.

[0055] The vehicle 10 also features a versatile assembly structure 35. The assembly structure 35 includes a center console, a dashboard, and an instrument panel. The assembly structure 35 includes a connector 36 – for example, a USB connector – and a visual display 37, such as a touch-sensitive human-machine interface (HMI).

[0056] Furthermore, the vehicle 10 has a sensor subsystem 60, which contains sensors that provide information for the controller system 20. The sensor input to the controller 20 is in Fig. Figure 2 shows a schematic diagram on the right side under the vehicle's hood. Exemplary sensors with the basic reference number 60 (601, 602, etc.) are also shown.

[0057] Sensor data refers to features such as vehicle operations, a vehicle position and a vehicle pose, passenger characteristics such as biometrics or physiological measurements, and environmental characteristics relating to a vehicle interior or an external environment of the vehicle 10.

[0058] Exemplary sensors include a camera 601 positioned in a rearview mirror of the vehicle 10, a vehicle sky or ceiling camera 602 positioned in a head area of ​​the vehicle 10, a world-facing camera 603 (pointing away from the vehicle 10), and a world-facing distance sensor 604.

[0059] World-facing sensors 603, 604 detect properties about an environment 11, which includes, for example, billboards, buildings, other vehicles, traffic signs, traffic lights, pedestrians, objects within range of the sensor, etc.

[0060] The aforementioned OBDs can be considered in various embodiments as local devices, sensors of subsystem 60, or both.

[0061] Local devices 34 (e.g., a passenger's telephone, a passenger's wearable device, or a passenger plug-in device) can also be considered sensors 60, for example, in embodiments where the vehicle 10 uses data provided by the local device based on an output from one or more sensors of the local device. For example, the vehicle system can use data from a user's smartphone displaying physiological data of the passenger, captured by a biometric sensor in the phone.

[0062] Furthermore, the vehicle contains 10 passenger compartment output components 70, such as loudspeakers and an instrument panel. The output components may also include a display screen on the dashboard or center console, a rearview mirror screen (for displaying images from a rear-view camera of the vehicle), and any visual display device 37 of the vehicle. III. Onboard computing architecture - Fig. 2

[0063] Fig. Figure 2 illustrates in greater detail the hardware-based computing or controller system 20 of Fig. 1. The controller system 20 may be referred to by other terms, such as computing device, controller, controller device or such descriptive terms, and it may be or contain one or more microcontrollers as mentioned above.

[0064] The controller system 20 is, in various embodiments, part of the aforementioned larger system 10, such as a vehicle.

[0065] The controller system 20 includes a hardware-based, computer-readable storage medium or data storage device 104 and a hardware-based processing unit 106. The processing unit 106 is connected or connectable to the computer-readable storage device 104 by means of a communication coupling 108, for example by means of a computer bus or by means of wireless components.

[0066] The processing unit 106 may be referred to by other names, such as processor, processing hardware unit, the like, or others.

[0067] The Processing Unit 106 can contain or consist of multiple processors, which can include distributed processors or parallel processes in a single machine or across multiple machines. The Processing Unit 106 can be used to support a virtual processing environment.

[0068] The processing unit 106 may, for example, contain a state machine, an application-specific integrated circuit (ASIC), or a programmable gate array (PGA) comprising a field PGA. References made here to the processing unit executing code or instructions to perform operations, actions, tasks, functions, steps, or the like may include the processing unit directly executing the operations and / or enabling the control or cooperation of another device or component to perform the operations.

[0069] In various embodiments, the data storage device 104 is any medium consisting of a volatile medium, a non-volatile medium, a removable medium and a non-removable medium.

[0070] The term computer-readable media and variants thereof, as used in the description and in the claims, refer to specific storage media. Fig. Figure 2 illustrates the hardware-based computing or controller system 20 of Fig. 1 in greater detail. The controller system 20 may be referred to by other terms, such as computing device, controller, controller device or a similar descriptive term, and it may consist of or contain one or more microcontrollers, as mentioned above.

[0071] The controller system 20 is, in various embodiments, part of the aforementioned larger system 10, such as a vehicle.

[0072] The controller system 20 includes a hardware-based computer-readable storage medium or data storage device 104 and a hardware-based processing unit 106. The processing unit 106 is connected or connectable to the computer-readable storage device 104 by means of a communication coupling 108, for example by means of a computer bus or by means of wireless components.

[0073] The processing unit 106 may be referred to by other names, such as processor, processing hardware unit, the like, or others.

[0074] The Processing Unit 106 can consist of or contain multiple processors, which may include distributed or parallel processors in a single machine or across multiple machines. The Processing Unit 106 can be used to support a virtual processing environment.

[0075] The processing unit 106 may, for example, comprise a state machine, an application-specific integrated circuit (ASIC), or a programmable gate array (PGA), including a field PGA. References made here to the processing unit executing code or instructions to perform operations, actions, tasks, functions, steps, or the like may include the processing unit directly executing the operations and / or enabling the control or cooperation of another device or component to perform the operations.

[0076] In various embodiments, the data storage device 104 is any medium consisting of a volatile medium, a non-volatile medium, a removable medium and a non-removable medium.

[0077] The term "computer-readable media" and its variants, as used in the description and claims, refer to specific storage media. The media can be a device and they can be permanent.

[0078] In various embodiments, the storage media include volatile and / or non-volatile, removable and / or non-removable media such as random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), semiconductor memory or other storage technology, a CD-ROM, a DVD, a BLU-RAY or other optical disk storage device, a magnetic tape, a magnetic disk storage device or other magnetic storage devices.

[0079] The data storage device 104 contains one or more memory modules 110 that store computer-readable code or instructions that can be executed by the processing unit 106 to perform the functions of the controller system 20 described herein. The modules and functions are described below in conjunction with Fig. 3 - 4 further described.

[0080] The data storage device 104 also includes, in various embodiments, auxiliary or support components 112, such as additional software and / or data that support the execution of the processes of the present disclosure, such as one or more passenger profiles or a group of standard and / or passenger-set preferences.

[0081] As mentioned, the controller system 20 also includes a communication subsystem 30 for communicating with local and external devices and networks 34, 40, 50. The communication subsystem 30 includes, in various embodiments, any wired input / output (I / O) 116, at least one long-range wireless transmitter / receiver 118, and one or more short- and / or medium-range wireless transmitters / receivers 120. A component 122 is shown as an example to emphasize that the system can be configured to accommodate one or more other types of wired or wireless communications.

[0082] The long-range transmitter / receiver 118 is designed in various embodiments to enable communications between the controller system 20 and a satellite and / or a mobile telecommunications network, which can be understood as also schematically indicated by the reference numeral 40.

[0083] The short- or medium-range transmitter / receiver 120 is designed to enable short- or medium-range communications, such as communications with other vehicles, vehicle-to-vehicle (V2V) communications, and communications with a transport system infrastructure (V2I) communications. More broadly, the term vehicle-to-entity (V2X) can refer to short-range communications with any type of external entity (e.g., devices assigned to pedestrians or cyclists, etc.).

[0084] For communication with V2V, V2I, or other devices outside the vehicle, such as local communication routers, etc., the transmitter / receiver 120 can be configured for short- or medium-range communications to communicate using one or more short- or medium-range communication protocols. Example protocols include Dedicated Short-Range Communications (DSRC), Wi-Fi®, Bluetooth®, infrared, Infrared Data Association (IRDA), Near Field Communications (NFC), the like, or improvements thereof (Wi-Fi is a registered trademark of the Wi-Fi Alliance of Austin, Texas; Bluetooth is a registered trademark of Bluetooth SIG, Inc., of Bellevue, Washington).

[0085] Via short, medium and / or long range wireless communications, the controller system 20 can send and receive information to and from the communication network 40 by operating the processor 106, for example in the form of messages or packaged data.

[0086] Remote devices 50, with which the subsystem 30 communicates, are located in various embodiments near the vehicle 10, away from the vehicle 10, or both.

[0087] The remote devices 50 can be configured with any suitable structure to perform the operations described herein. An exemplary structure includes any or all structures such as those described in connection with the vehicle's computing device 20. For example, a remote device 50 includes a processing unit, a storage medium containing modules, a communication bus, and an input / output communication structure. These features are understood to be defined for the remote device 50 by Fig. 1 and are shown by the cross-reference provided by this paragraph.

[0088] Local devices 34 are indeed in Fig. 1 and Fig. 2 are shown inside vehicle 10, however any one of these can be outside the vehicle and in communication with the vehicle.

[0089] Examples of remote systems 50 include a remote server (for example, an application server), or a remote data center, a customer service center, and / or a control center. A passenger's computing device or electronic device 34, such as a smartphone, may also be located remotely from the vehicle 10 and communicate with the subsystem 30, for example, via the internet or another communication network 40.

[0090] An example of a control center is the OnStar® control center, which features facilities for interacting with vehicles and passengers via long-range communications, such as satellite or cellular telecommunications, either through the vehicle or by other means (for example, via a mobile phone). ONSTAR is a registered trademark of the OnStar Corporation, a subsidiary of General Motors Company.

[0091] As mentioned, the vehicle 10 also includes a sensor subsystem 60, which contains sensors that provide information to the controller system 20 regarding objects such as vehicle operations, vehicle position, vehicle pose, passenger characteristics such as biometrics or physiological measurements, and / or the environment around the vehicle 10. The arrangement can be configured such that the controller system 20 communicates with sensors of the sensor subsystem 60 via short-range wired or wireless communication links 116, 120, or at least receives signals from them.

[0092] In various embodiments, the sensor subsystem 60 includes at least one camera and at least one distance sensor 604, such as radar or sonar, which are directed away from the vehicle, for example to support autonomous driving.

[0093] Visible light cameras 603, directed away from the vehicle 10, may include a monocular forward-facing camera, such as those used in lane departure warning (LDW) systems. Embodiments may include other camera technologies, such as a stereo camera or a trifocal camera.

[0094] Sensors designed to detect external conditions can be arranged or oriented in any variety of directions without departing from the scope of this disclosure. For example, the cameras 603 and the distance sensor 604 can be oriented in any or a selected position from (i) forward from a front center point of the vehicle 10, (ii) rearward from a rear center point of the vehicle 10, (iii) laterally to the vehicle 10 from a side position, and / or (iv) between these directions, each directed, for example, at or toward any height.

[0095] The 604 distance sensor can, for example, include a short-range radar (SRR), an ultrasonic sensor, a long-range radar such as those used in autonomous systems or adaptive cruise control (ACC) systems, sonar, or a light detection and distance sensor (LiDAR sensor).

[0096] Other exemplary sensor subsystems 60 include passenger compartment sensors 601, 602, which are designed and arranged (e.g., positioned and fitted into the vehicle) to detect activities, persons, passenger compartment environmental conditions, or other features relating to the interior of the vehicle. Exemplary passenger compartment sensors 601, 602 include microphones, in-vehicle visible light cameras, seat weight sensors, salinity, retinal or other user characteristics, biometrics or physiological measurements, and / or the environment surrounding the vehicle 10.

[0097] The passenger compartment sensors (601, 602, etc.) of the vehicle sensors 60 can comprise one or more temperature-sensitive cameras or sensors. As mentioned, an exemplary temperature sensor is a thermographic camera, a thermal imaging camera, or an infrared camera arranged in the vehicle 10 to detect temperature conditions within the vehicle and, in particular, the temperature conditions of occupants. In various embodiments, the temperature cameras in the vehicle 10 are preferably positioned at a high location. Exemplary locations include on a rearview mirror and in a vehicle headliner compartment.

[0098] A higher positioning reduces interference from lateral obstacles, such as front-row seatbacks that block passengers in the second or third row, or obstruct larger parts of those passengers. A higher-positioned temperature camera, or any type of camera, will likely be able to capture the temperature of a larger portion of each passenger's body—for example, torso, legs, and feet.

[0099] Other exemplary sensor subsystems 60 include dynamic vehicle sensors 134, such as an inertial measurement unit (IMU) which includes one or more accelerometers, a wheel sensor, or a sensor connected to a steering system (e.g., a steering wheel) of the vehicle 10.

[0100] The Sensors 60 can contain any sensor for measuring a vehicle pose or other dynamics, such as position, speed, acceleration or height - e.g. a vehicle height sensor.

[0101] The sensors 60 can include any known sensor for measuring the vehicle's environment, including those mentioned above, and others, such as a precipitation sensor to detect whether and how much it is raining or snowing, a temperature sensor, and any others.

[0102] Sensors for capturing user characteristics include any biometric sensor, such as a camera used for retinal detection or detection of other eye features, facial recognition, fingerprint recognition, a temperature sensor, a microphone used for speech recognition or other user identification, other types of camera-based user identification systems, a weight sensor, a salinity sensor, respiratory quality sensors (e.g., a breathalyzer), a user temperature sensor, an electrocardiogram sensor (ECG sensor), electrodermal activity (EDA sensors) or galvanic skin response (GSR) sensors, blood volume pulse sensors (BVP sensors), heart rate (HR sensors), an electroencephalogram sensor (EEG sensor), an electromyography sensor (EMG sensor), and user temperature, a salinity sensor, the like, or others.

[0103] Interfaces between user and vehicle, such as a touch-sensitive display 37, push buttons, knobs, the like or others, can also be considered part of the sensor subsystem 60.

[0104] Fig. Figure 2 also shows the passenger compartment output components 70 mentioned above. In various embodiments, the output components include a mechanism for communicating with vehicle occupants. The components include, but are not limited to, loudspeakers 140, visual displays 142, such as the instrument panel, the center console display screen, and the rearview mirror screen, and haptic outputs 144, such as vibration actuators for the steering wheel or the seat. The fourth element 146 in this Section 70 is provided to emphasize that the vehicle may include any of a wide variety of other output components, such as components that provide aroma or light in the passenger compartment. IV. Additional vehicle components - Fig. 3

[0105] Fig. Figure 3 shows an alternative view of vehicle 10 of Fig. 1 and Fig. 2, which highlights exemplary storage components and shows associated devices.

[0106] As mentioned, the data storage device 104 contains one or more modules 110 for carrying out the processes of this disclosure. The device 104 may also contain auxiliary components 112, such as additional software and / or data that support the execution of the processes of this disclosure. These auxiliary components 112 may, for example, include additional software and / or data that support the execution of the processes of this disclosure, such as one or more user profiles or a set of default and / or user-configured preferences.

[0107] Any code or instructions described here can be part of more than one module. And any functions described here can be executed by executing instructions in one or more modules, although the functions may be primarily described in connection with a module by means of a primary example. Each of the modules can be designated by any variety of names, such as a term or expression that indicates its function.

[0108] Submodules can cause the processing-based hardware unit 106 to execute specific operations or routines of module functions. Each submodule can also be designated by any variety of names, such as a term or expression that indicates its function.

[0109] Examples of modules shown (110) include: - an input interface module 302; - a database module 304; - a passenger attention module 306 of the autonomous vehicle; - a passenger positioning module 308 of the autonomous vehicle; - an object localization module 310; - a driver module 312 of the autonomous vehicle; - a context module 314 of autonomous driving; - an activity module 316; and - an output interface module 318.

[0110] Others in Fig. The vehicle components shown comprise the vehicle communication subsystem 30 and the vehicle sensor subsystem 60. These subsystems act, at least partially, as input sources for the modules 110 and specifically for the input interface module 302. Exemplary inputs from the communication subsystem 30 include identification signals from mobile devices, which can be used to identify or register a mobile device and thus the associated user for the vehicle 10, or to at least provisionally register the device / user, followed by a higher-level registration, such as biometric or passcode registration.

[0111] Examples of inputs from the vehicle's sensor subsystem 60 include, but are not limited to: - biometric sensors that provide biometric data regarding vehicle occupants, such as skin or body temperature for each occupant; - Input devices of vehicle occupants (human-machine interfaces (HMIs), such as a touch-sensitive screen, buttons, knobs, a microphone, etc.); - Passenger compartment sensors that provide data about properties inside the vehicle, such as the position, orientation or gaze of a user, the temperature in the vehicle interior, weight sensors in the seat and motion detection sensors; - Environmental sensors that provide data about conditions around a vehicle, such as an external camera and distance sensors - e.g. LiDAR, radar; and - sources separate from the vehicle 10, such as local devices 34, devices worn on the body by pedestrians, other systems in the vehicle such as portable user devices, a local infrastructure (local signal stations, mobile phone masts, etc.), satellite systems and remote systems 34 / 50, which provide any of a wide variety of information, such as user identification data, user history data, user selections or user preferences, contextual data (weather, road conditions, navigation, etc.).), program or system updates - remote systems may include, for example, application servers corresponding to the applications running on the vehicle 10 and on any relevant user devices 34, computers of a user or caregiver (parent, work supervisor), servers of vehicle operators, a customer service control center system, such as systems of the aforementioned OnStar® control center, or a vehicle operator system, such as that of a taxi company that operates a fleet to which the vehicle 10 belongs, or of a ride-sharing service operator.

[0112] The view also shows exemplary vehicle outputs 70 and user devices 34 that can be positioned in the vehicle 10. The outputs 70 include, but are not limited to: - Vehicle dynamic actuators, such as those relating to autonomous driving - vehicle brake, throttle valve, steering, etc.; - Vehicle climate actuators, such as those that control the temperature, humidity, zone outputs, and fan speeds of an HVAC system; and - local devices 34 and remote systems 34 / 50, for which the system can provide a wide variety of information, such as user identification data, user biometric data, user historical data, contextual data (weather, road conditions, etc.), instructions or data for use in providing notifications, alerts or messages to the user or to relevant entities such as authorities, first responders, parents, an operator or owner of a vehicle concerned 10 or a customer service center system, such as the OnStar® Control Center, to initiate communications such as a call or a text / SMS to first responders, parents, an operator, an owner, etc.

[0113] The modules, submodules and their functions are described in more detail below. V. Algorithms and Processes - Fig. 4V.A. Introduction to Algorithms

[0114] Fig. Figure 4 shows an exemplary algorithm, which is schematically represented by a process flow or algorithm 400 in accordance with embodiments of the present technology. Although a single process flow is shown for simplicity, any of the functions or operations in one or more processes, routines, or subroutines can be performed by one or more algorithms, devices, or systems.

[0115] It is understood that the steps, operations, or functions of Process or Algorithm 400 are not necessarily presented in any particular order, and that performing some or all of the operations in an alternative order is possible and considered. The processes can also be combined or overlap, so that one or more operations from one process are performed in another.

[0116] The operations are presented in the order shown for the sake of simplicity of description and illustration. Operations can be added, omitted, and / or performed simultaneously without departing from the scope of the attached claims. It is also understood that the illustrated processes 400 can be terminated at any time.

[0117] In certain embodiments, some or all of the operations of the processes 400 and / or substantially equivalent operations are performed by a computer processor, such as the hardware-based processing unit 106, which executes computer-executable instructions stored, for example, in a non-temporary computer-readable storage device, such as any of the data storage devices 104 or a mobile device as described above. VB System Components and Functions

[0118] Fig. 4 shows how the components of Fig. 3 interact in accordance with various exemplary process flows and algorithms 400.

[0119] The input interface module 302, which is executed by a processor, such as the hardware-based processing unit 106, receives any of a wide variety of input data or signals, including from the sources described in the previous section (IV.).

[0120] Input sources include the vehicle sensors 60 and the local or remote devices 34, 50 via the vehicle communication subsystem 30. Inputs also include a vehicle database via the database module 304.

[0121] After any formatting, conversion or further processing at the input interface module 302, input data is forwarded to the activity module 316 or first to other modules.

[0122] In various implementations, the activity module 316 requests (pull by activity module) relevant data from the database module 304, receives it without a request (push from the database module), or obtains it in another way. The database module 304 can communicate with, contain, or be part of memory sections of the vehicle 10, such as a section that stores the aforementioned auxiliary data. As mentioned, the auxiliary data can contain one or more user profiles. These profiles may, for example, have been pre-generated by the system processor or received from a remote source such as server 50 or a remote user computer.

[0123] The profile for each user can contain user-specific preferences that have been communicated to the system by the user, for example via a touch-sensitive screen or a microphone interface.

[0124] Preferences encompass any settings that influence how the system interacts with the user or with a non-vehicle system, such as a server or user device (share data). Examples of preferences include volume, tone, or other sound preferences for media delivery, and the type or volume of notifications provided to the user.

[0125] Data from database module 304 can also contain historical data, representing, for example, past activity between the system and a user, between the system and other users, or between other systems and these or other users. As just one example, if the user reacts in a typical way to a specific situation (e.g., driving over a bridge) over repeated occurrences (looking at the water or the road below), this activity can be stored in connection with the user and used to determine how or when information should be presented to the user, such as via the vehicle's heads-up display (HUD).If a user turns down the volume in their acoustic zone when receiving a particular notification, the system can generate historical or learned data for that user requesting that the system use a lower volume for the notification.

[0126] User preferences can include, for example, preferences regarding how messages are delivered - e.g., HUD versus center console screen and with or without sound, and if with sound, with a preferred type of sound.

[0127] Preferences can be received from a remote profile or a non-vehicle profile, such as a profile stored on a user's mobile device 34 or a remote server 50, and features of local and remote profiles can be synchronized or shared between the vehicle 10 and the remote source.

[0128] The autonomous vehicle's passenger attention module 306 determines the gaze of one or more passengers, at least on the basis of sensor data indicating the direction in which one or both of a passenger's eyeballs are facing.

[0129] The determination can be based on any one of a wide variety of other conditions, such as the orientation or pose of a passenger's head, the passenger's position in vehicle 10, the orientation of the vehicle 10 in which the passenger is currently traveling, the pose of vehicle 10, and the location of vehicle 10. For example, the module can determine that the passenger's head is facing directly forward from a front-row passenger seat, and that the passenger's eyes are directed 30 degrees to the right and 10 degrees vertically above the passenger's eye plane.

[0130] Determining the attention or gaze of the passenger can be performed for each of several passengers.

[0131] The autonomous vehicle's passenger positioning module 308 can be used to determine the passenger's position or pose within the vehicle. Determining the passenger's pose and / or position can be performed for each of several passengers.

[0132] Position determination can be based on one or more passenger compartment cameras and on one or more other passenger compartment sensors, such as a temperature sensor, a short-range radar, etc.

[0133] In various embodiments, the position or pose data of the passenger are based instead on, or also on, measurements from a mobile device 34, such as a tablet 33 or smart glasses 32.

[0134] Components of mobile devices used for this function include, to name just a few examples, a front-facing camera, a rear-facing camera, distance sensors, and an inertial measurement unit (IMU) incorporating one or more accelerometers. The measurement can be performed for multiple passengers, generally at the same time, or sequentially.

[0135] The object localization module 310 determines the location of objects outside the vehicle 10. In various embodiments, this determination is based on dynamic real-time data received from one or more world-facing cameras 603 and / or from one or more world-facing distance sensors 604.

[0136] In various embodiments, the object localization data can be received instead from an image or navigation database or a program.

[0137] The autonomous vehicle's driver module 312 detects whether a condition exists that allows a passenger to take manual control of the vehicle 10. Based on the system programming, the condition may require that the passenger be recommended to take control in some situations, and that in other situations it is more urgently requested that manual control be taken over, or even more urgently instructed that the user switch to manual mode or drive out and park.

[0138] The autonomous driving context module 314 generates or obtains data indicating other relevant circumstances to be used by the activity module 316 to determine how to act.

[0139] For example, the context can include a passenger's plan or itinerary. The itinerary might indicate that the user is planning to attend an event at a local theater and has specifically reserved tickets that are waiting for them at the pick-up counter. The context data can contain these facts and the location of the pick-up counter. Alternatively, the system—for example, context module 314—can obtain the pick-up counter location based on these facts, perhaps by retrieving the location from a remote server 50.

[0140] Based on the various inputs, the activity module 316 performs one or more of the potentially resulting operations that have been explicitly described and referenced or otherwise suggested here.

[0141] In various embodiments, operations of activity module 316 include, to name just a few examples, one or more of the following: - Determining a communication to be provided to the passenger via a vehicle HUD, vehicle speakers, vehicle lights or a screen, or via an output interface of the mobile device. The communication may include presenting a virtual or augmented reality view, such as highlighting in real time a pedestrian or other object dynamically detected by one or more vehicle sensors that the passenger apparently did not perceive based on their recorded gaze. The communication may be designed to instruct the passenger that manual operation of the vehicle is recommended or necessary. The determination that manual operation is recommended or necessary may be based on a predefined setting or user preference.Communication can be generated in response to the detection, based on any of the aforementioned vehicle sensors and / or mobile device sensors, that passengers do not exhibit a position, pose, and / or gaze indicating their readiness to assume manual control. In various implementations, this communication includes information regarding a destination indicated by the passenger or their travel path. Module 314 can determine that a notification is displayed on a HUD highlighting the location of a pick-up window at a local theater to which the vehicle is currently autonomously driving, ensuring the passenger knows precisely where to go when the vehicle stops and the passenger disembarks. - Determining how the autonomous driving of the vehicle should be adjusted, for example by adjusting a throttle valve, brakes or steering gear. - Determine that an advertising account is credited, or send a message to a remote system (e.g., an ad server) in response to providing a communication to the passenger to direct the passenger's attention to an advertising object (e.g., billboard, storefront, etc.), and detect that the passenger has actually changed their gaze to focus on the object. An ad arrangement may include interactions with an advertising entity that pays for reported views of advertisements. - Determine how to communicate with the passenger to motivate them to assume a position of manual control of the autonomous vehicle. This communication can be provided via a human-machine interface (HMI) of the autonomous vehicle, with the alert received by the passenger intended to motivate them to take over manual control of the autonomous vehicle. - Determine whether to deliver further advertisements to the user, or purchase options, a communication, in response to the passenger looking at an advertising object (e.g. billboard or shop front), either in response to whether the gaze is directed there at all, or in response to the system causing it to do so, or that the gaze remains on the object for more than a threshold time period; and - Comparing or correlating a passenger's visibility with a passenger's gaze and using the results of the comparison or correlation, for example in determining a way in which augmented or virtual reality data should be presented via a HUD or mobile device 34.

[0142] The output interface module 318 formats, converts, or otherwise processes output from the activity module 316 before delivering it to the various output components for implementation.

[0143] As shown, exemplary output components of the system include speakers, screens, or other outputs of the vehicle.

[0144] Exemplary output components of the system can also include remote systems, such as remote servers and user computer systems (e.g., a home computer). The output can be received and processed by these systems, for example, to update a user profile with a determined preference, an activity undertaken with regard to the user, or similar or other information.

[0145] In embodiments where the passenger wears or has a mobile or wearable device 34, such as a tablet 33 or glasses 32, the output communication can be provided by the mobile device—e.g., highlighting a distant object in a virtual / augmented reality display on the mobile device's screen, and / or providing audio output (tones, synthetic speech, etc.). The same mobile device can therefore be used both as an input source, e.g., to determine the passenger's gaze, pose, or position, etc., and as an output source for a destination or waypoint, for example, if the user's phone also provides output to another system, such as a remote server 50.

[0146] Exemplary output components of the system can also include a vehicle database. Output data can be provided to the database module 304, which can be used, for example, during system training, and can be stored in a suitable user account of the auxiliary data 112. VI. Additional Structures, Algorithm Features and Operations

[0147] In combination with any of the other embodiments described herein, or instead of any embodiments, the present technology can incorporate any structure or perform any functions as follows. i. The technology includes systems and methods designed in various embodiments to connect the passenger's attention with points of interest in the driving or commuting scene (e.g., display boards, pedestrians, tourist attractions), making advantageous use of embedded external and internal sensing technology, specifically but not limited to application in autonomous vehicles. ii. The technology includes systems and methods designed in various embodiments to perform automatic monitoring of points or objects of interest outside the vehicle in a driving and commuting scene on which the passenger's attention is or has recently been focused, based on any of various factors such as the passenger's gaze, the passenger's pose, a position, a vehicle orientation, a vehicle position, and a position of objects. iii. The technology may include an advertising-based arrangement in which credits are awarded in response to the user's attention being focused on or directed to advertising objects. iv. The technology can be used in tour applications or sightseeing applications. v. In various embodiments, if the passenger's gaze is on or likely to be very close to an object, the system will not communicate with the user to highlight or call out the object in order to avoid overtaxing the passenger's senses. vi. The technology may include a system that can automatically monitor which objects or points of interest (for example, in an environment outside the passenger compartment) attract the attention of the passenger / driver, and, in various implementations, which points or objects attract attention under which circumstances—for example, billboards, which are perceived much more readily when the occupant is in the driver's seat than when seated in a front or rear passenger seat. The system is designed to provide appropriate alerts or communications to the passenger based on the context and, in some implementations, for safety reasons. vii. The technology advantageously utilizes advanced external sensors already provided in many, if not most, autonomous vehicles to dynamically determine in real time the objects of interest to the driver or passengers (e.g., where their gaze is or was recently directed) and to dynamically allocate the passenger's attention to the object in the system, for example, for subsequent use, such as crediting a user account for viewing an advertisement, or determining an output action, such as highlighting the object or presenting information about the object, for example, using virtual reality on a vehicle window, smart glasses, or another user device, a hologram, etc. viii. The technology uses, in various embodiments, one or more of three form factors or settings, wherein the system determines any position / pose / orientation of a passenger and, in some cases, actively monitors it, using one or more of: 1. A portable device – e.g., a tablet computer or a smartphone. As mentioned, the portable device can also be an output. 2. A device worn on the body, such as wearable eyeglasses. As mentioned, the glasses can also be an expense. 3. an eye-tracking device embedded in the passenger compartment, facing the user. ix. Regarding the form factor of the tablet or other portable device, most tablets have a world-facing (e.g., rear-facing) camera (e.g., RGB or RGB-D). They also have a user-facing camera (e.g., a front-side camera). The cameras may be RGB-D cameras, for example. Other tablet sensors may be arranged on the tablet to detect objects in the same or different directions. In some autonomous driving systems, the systems are programmed to operate in accordance with an arrangement of different levels of automation, with, for example, Level 1 autonomous driving involving only a lightly automated function, such as maintaining the automatic following distance of the automatic cruise control. Higher levels, such as Level 5 or 6, may involve full automation.At intermediate levels of automation, such as when the vehicle is operating at Level 3, the passenger often has their hands free or off the steering wheel, but not always. Nevertheless, hands-free driving, even when used occasionally, can be beneficial, as the occupant can hold and focus on a user device such as a tablet. The vehicle would instruct the user when it is time to take over the steering wheel. Using a tablet can offer several advantages in the context of active safety and advanced driver assistance systems (ADAS), such as providing the passenger with information (e.g., via the tablet) relevant to what they see while traveling along a planned route.The tablet can, on the one hand, provide the passenger's point of view using a rear-facing camera, provided the passenger holds the tablet in such a way that it is generally oriented in the same direction and / or at the same point of view as the user's gaze. On the other hand, the system can display relevant information aligned with the passenger's point of view – for example, by highlighting it or displaying it via a head-up display (HUD). x. With regard to the vehicle's form factor, the system can, in various embodiments, generate a widescreen HUD with augmented reality and utilize a passenger-facing, frontal 3D camera or other sensor within the vehicle, which can be designed and positioned to function as an eye-tracking device. Through sensors embedded in the vehicle, the system can detect objects of interest within the passenger's field of vision. The system can monitor the objects seen by the passenger by calculating a real-time view of the user or a projection of that view onto the objects in the scene. xi. Two or three form factors can be used at once - e.g., the input can include vehicle sensors and tablet sensors, and the output can be provided for vehicle outputs (screen, HUD, speakers, etc.) and / or for display on smart glasses. xii. The system can, in various embodiments, depend on the sensors on the tablet that are facing the user in order to monitor the passenger's gaze. xiii. Furthermore, with regard to the form factor of the glasses, the system can utilize the glasses' augmented reality functions. This factor is useful, for example, for noting a route or objects in a scene while the vehicle is in motion. This notation can therefore be done comprehensively using AR glasses worn by the passenger. If the AR glasses have an eye-tracking device, the system can observe what is actually visible from the passenger's point of view and what they actually see, which may not necessarily be the same.The system can therefore advantageously utilize both the user-facing 3D camera on the tablet for eye tracking and the tablet's rear-facing camera to record / play back relevant virtual artifacts or to present information to the passenger, tailored to their perspective within their context / frame of reference / reality. Thus, the system can effectively use an existing eye-tracking source (e.g., in smart glasses) in addition to the output from a world-facing rear-facing 3D camera or wearable glasses to accurately reproduce / record relevant information for the passenger. In any case, the system can be designed to correlate what the passenger actually sees with what is visible from their perspective or viewpoint, for example, through synchronization with sensors embedded in the vehicle. xiv. Various embodiments of the technology have been disclosed, taking into account that passengers (which include the drivers in semi- or fully autonomous driving) will perceive their surroundings in a totally different way in autonomous driving than in manual driving, since automation will allow the passenger (including the driver) to concentrate on activities other than driving. xv. In various embodiments, the system benefits from specific information indicating which objects in the commuter scene (outside the passenger compartment) have, have recently had, or are attracting the passenger's attention. xvi. Furthermore, with regard to active safety applications, the system is designed to highlight objects, such as pedestrians or a ball rolling into the road, using augmented reality techniques. In various embodiments, this is specifically implemented for objects that have not yet attracted the attention of the occupants (e.g., the driver or passengers), in order to keep the occupants' attention focused on the object. xvii. The system is designed in various embodiments to ensure that objects which already have the attention of an occupant (e.g., a driver who is driving fully or partially manually) are not highlighted. This attention, in turn, may be based, for example, on the occupant's position, the user's gaze, or the occupant's pose (e.g., a head position), etc. The cognitive challenge for the occupant is thus intentionally limited, allowing the occupant to concentrate on other things. xviii. The system can be used in sightseeing or tour situations, for example by highlighting points of interest (POIs), such as on a HUD or user tablet, which passengers are encouraged to monitor during the journey. These points of interest include areas in front of, adjacent to, or around the vehicle, or an object in front of, adjacent to, or around the vehicle (e.g., historical buildings on a tour with autonomous vehicles), or other local tourist attractions. If the passenger has not noticed an area or object of interest, the system can provide a timely notification, such as a text message or a visual cue, to draw the passenger's attention to the area, object, or point of interest (POI).The system can be designed to provide information about an object, area, or point of interest (POI) in response to the detection that the passenger is already looking at it, or that the passenger has changed their gaze towards it, in response to the system prompting the passenger to look at it, for example, by highlighting the HUD, displaying another AR indicator, providing voice notifications, etc. xix. In some cases, the system is designed to use a tablet, smart glasses, or other suitable user device belonging to the passenger as a source of input and / or output, as mentioned, together with or instead of one or more suitable vehicle sensors. The use of a portable device may provide the user with greater flexibility or otherwise a better user experience, such as by allowing the passenger, when using a tablet or smart glasses, considerable or complete freedom to look at the tablet, or to hold it in an orientation not aligned with the direction of travel, or to hold or look at the tablet in any desired direction at any given time.In various implementations, the system correlates world axes or a reference frame of the sensors embedded in the vehicle with a reference frame of the output, or vice versa. In various embodiments, such as those involving the use of a tablet and / or smart glasses, the system advantageously utilizes simultaneous localization and mapping (SLAM) technology. For example, if the system incorporates or uses eye-tracking technology, it can obtain accurate information regarding conditions such as the orientation of the passenger's head and gaze within a context that includes detected or otherwise known objects, areas, or points of view (POLs) (e.g., from map data). SLAM technology can also assist in embodiments where the occupant uses a tablet, for example, by using SLAM technology to align or correlate (i) the world as seen by the tablet with (ii) the world as seen by the vehicle.Correlation can be used to ensure, or better yet, to ensure, that an output provided to the user, such as an AR display, is occupant- or human-centered. VII. Selected Advantages

[0148] Many of the advantages and benefits of this technology have been described above. This section reiterates some of these and refers to others. The benefits described are not exhaustive.

[0149] Safety is improved because passengers in autonomous vehicles are notified by the system that they need to get into position and / or focus their attention on manual driving when required.

[0150] Safety is further enhanced by dynamic, real-time communications provided to passengers regarding objects of interest, such as a pedestrian crossing the street. This function is activated in some cases even when the system detects that the passenger is not currently looking at the object.

[0151] The system can provide the benefit of earning money, for example by using the advertising system described.

[0152] A further benefit of our proposed solution is that by (i) correlating a passenger's view with what the passenger sees and / or (ii) correlating vehicle viewpoints with a portable user device, the system ensures that the output is human-centered. One way to achieve this is to focus the extension on the user's / passenger's attention—that is, to reproduce a passenger viewpoint with relevant virtual artifacts that are aligned with the passenger's attention.

[0153] In operation, the technology improves the satisfaction of the driver and / or passenger, including well-being, of using automated driving by adjusting any wide variety of vehicle and / or non-vehicle characteristics, such as driving style parameters of the vehicle.

[0154] The technology will increase the use of automated driving systems, as users are more likely to utilize or learn a vehicle's capabilities for more advanced autonomous driving when they become aware of them, and especially when using features that enable and / or enhance the autonomous driving experience.

[0155] In the same way, a "relationship" between users and a vehicle can be improved—the user will view the vehicle more as a familiar tool, an assistant, or a friend. Consequently, the technology can also influence acceptance levels and, relatedly, the marketing and sales of vehicles capable of automated driving by increasing the confidence of users or potential users in autonomous driving systems. And people are more likely to use or buy a vehicle capable of autonomous driving, to buy another one, or, under these circumstances, to recommend it to others or set an example for others in its use.

[0156] Another benefit of using the system is that users need to invest little or no effort in setting or calibrating some, most, or even all parameters of an automated driving style, as these are automatically set or adjusted by the system. This is achieved, for example, by learning about the behavior of occupants, their preferences, and so on during use. This minimizes stress for the user and thus increases user satisfaction and well-being with the autonomous driving vehicle and its functionality. VIII. Conclusion

[0157] Various embodiments of the present disclosure are disclosed here. The disclosed embodiments are merely examples, which can be implemented in different and alternative forms and combinations thereof.

[0158] The embodiments described above are merely exemplary illustrations of implementations that have been disclosed for the purpose of clearly understanding the principles of the disclosure.

[0159] References made here to how a feature is arranged may refer to, but are not limited to, how the feature is positioned in relation to other features. References made here to how a feature is designed may refer to, but are not limited to, how the feature is dimensioned, shaped, and / or made of material. For the sake of simplicity, the term "designed" may be used to refer to both the design and the arrangement described above in this paragraph.

[0160] References to direction are provided here primarily to facilitate description and simplify the description of the exemplary drawings, and the temperature management systems described can be implemented in any of a wide variety of orientations. References to direction shown here are not intended to be restrictive. For example, references to above, below, top, bottom, or side are not provided to limit the ways in which the technology of this disclosure may be implemented. While a top surface is designated, the designated surface may, for example, but need not, be vertically upward or at the top in a design, production, or operating frame of reference.In various embodiments, the surface can instead be located, for example, next to or below other components of the system.

[0161] Each component shown or described in the figures as a single object can be replaced by several such objects, each designed to perform the functions of the single described object. Similarly, any number of objects can be replaced by a single object designed to perform the functions of the multiple described objects.

[0162] Variations, modifications, and combinations of the embodiments described above can be performed without departing from the scope of the claims. All such variations, modifications, and combinations are contained herein by the scope of this disclosure and the following claims.

Claims

[1] System for use with an autonomous vehicle (10), comprising: a non-temporary storage device (104), comprising: a context module (314) of the autonomous vehicle (10) which, when executed by a hardware-based processing unit (106), determines whether manual operation of the vehicle (10) is recommended or required during autonomous operation of the vehicle (10); and a driver module (312) of the autonomous vehicle (10) which, when executed by the hardware-based processing unit (106), determines, based on occupant position data received from an occupant position sensor, that a vehicle occupant is not ready to take manual control of the autonomous vehicle (10); wherein the occupant position sensor is part of a portable occupant device (31, 32, 33) which can be used both as an input source and for output communication. [2] System according to claim 1, wherein: The occupant position data displays the position of several vehicle occupants of the vehicle (10); and the driver module (312) of the autonomous vehicle (10), which, when executed by the hardware-based processing unit (106), determines, based on the occupant position data received from an occupant position sensor, that none of the vehicle occupants are willing to take manual control of the autonomous vehicle (10). [3] System according to claim 2, wherein the driver module (312) of the autonomous vehicle (10), when executed by the hardware-based processing unit (106), determines which of the multiple vehicle occupants is in a best position to take over manual control of the autonomous vehicle (10). [4] System according to claim 1, wherein the non-temporary storage module comprises an activity module which, when executed by the processing unit (106), determines an action to be taken in response to the finding that the occupant is not ready to assume manual control of the autonomous vehicle (10), wherein the action comprises providing occupant communication designed to motivate the occupant to assume a position to assume manual control of the autonomous vehicle (10). [5] System according to claim 1, wherein the non-temporary storage device (104) comprises an activity module (316) which, when executed by the processing unit (106), determines an action to be taken in response to the finding that the occupant is not ready to assume manual control of the autonomous vehicle (10), the action comprising adjusting the autonomous driving operation of the vehicle (10). [6] System according to claim 1, wherein the context module (314) of the autonomous vehicle (10), when executed to determine that manual operation of the vehicle (10) is recommended or necessary during autonomous operation of the vehicle (10), determines that manual operation is recommended or necessary based on a pre-established occupant preference setting. [7] System for use with an autonomous vehicle (10), comprising: a non-temporary storage device (104), comprising: a context module (314) of the autonomous vehicle (10) which, when executed by a hardware-based processing unit (106), determines whether manual operation of the vehicle (10) is recommended or required during autonomous operation of the vehicle (10); and a driver module (312) of the autonomous vehicle (10) which, when executed by the hardware-based processing unit (106), determines, based on occupant gaze data showing a glance of the vehicle occupant, that a vehicle occupant is not ready to take manual control of the autonomous vehicle (10); wherein the occupant gaze data are received by a portable occupant device (31, 33) or by a body-worn device (32) of the occupant wherein an occupant position sensor is part of the portable occupant device (31, 32, 33) which can be used both as an input source and for output communication.

Images