Driver-initiated vehicle component actuation

Abstract

A system includes a computer having a processor and memory. The memory stores instructions executable by the processor for determining a passenger presence, determining a vehicle operator's gaze direction based on an image of the vehicle operator, and determining a position of a vehicle operator's hand based on the image of the vehicle operator. The system actuates a vehicle component based on the determined passenger presence, the determined vehicle operator's gaze direction, and the determined position of the vehicle operator's hand.

Description

FIELD OF TECHNOLOGY

[0001] The present description generally relates to vehicle component actuation and, more particularly, relates to techniques for controlling vehicle component actuation in response to determining whether a vehicle operator's hands are in contact with a vehicle steering wheel and the vehicle operator's gaze is in a forward direction with respect to the vehicle. GENERAL STATE OF THE ART

[0002] A vehicle may include a human-machine interface (HMI), such as a touchscreen display associated with, for example, one or more of an infotainment system, a vehicle navigation system, and / or a communications system. A user, such as a passenger in the vehicle, may provide input to these systems via the HMI. When the vehicle is in motion, the HMI may be deactivated with respect to certain systems and / or features of those systems. SUMMARY

[0003] In one aspect, the present disclosure describes a system including a computer having a processor and a memory. The memory includes instructions executable by the processor for determining a presence of a passenger, determining a gaze direction of a vehicle operator based on an image of the vehicle operator, and determining a position of a hand of the vehicle operator based on the image of the vehicle operator. The system actuates a vehicle component based on the determined presence of the passenger, the determined gaze direction of the vehicle operator, and the determined position of the vehicle operator's hand.

[0004] In another aspect, it describes that the vehicle component may be a steering component of a vehicle and the instructions may include instructions for actuating the steering component when the position of the hand is not in contact with a steering element of the vehicle and the instantaneous direction is away from a forward direction with respect to the vehicle.

[0005] In another aspect, it describes that the vehicle component may be a human-machine interface (HMI) of a vehicle and the instructions may include instructions for activating the HMI when the passenger is present, the position of the hand is in contact with a steering element of the vehicle, and the eye direction is in a forward direction with respect to the vehicle.

[0006] In another aspect, it describes that the instructions may include instructions to deactivate the HMI when the position of the hand is not in contact with the steering element of the vehicle or the eye direction is not in the forward direction.

[0007] In another aspect, it describes that the instructions for determining a gaze direction may include instructions for using the image of the vehicle driver to determine a view angle and map the view angle to a direction relative to a vehicle.

[0008] In another aspect, it describes that the instructions for determining a position of a hand of the vehicle operator may include instructions for using the image of the vehicle operator to create a node model of the vehicle operator.

[0009] In another aspect, it describes that the instructions may include instructions for determining a gaze direction of the passenger based on an image of the passenger and determining a position of a hand of the passenger based on the image of the passenger.

[0010] In another aspect, it describes that the vehicle component may be a vehicle's HMI, and the instructions may include instructions for activating the HMI when the passenger is present, the position of the vehicle operator's hand is in contact with a steering element of the vehicle, the vehicle operator's eye-direction is in a forward direction relative to the vehicle, the passenger's eye-direction is toward the HMI, and the position of the passenger's hand is in contact with the HMI. The instructions may include instructions for deactivating the HMI when the passenger is not present. The HMI may be a vehicle navigation system.

[0011] In yet another aspect, it describes a method that includes determining a passenger's presence, determining a vehicle operator's gaze direction based on an image of the vehicle operator, and determining a position of a vehicle operator's hand based on the image of the vehicle operator. The method may include actuating a vehicle component based on the determined passenger's presence, the determined vehicle operator's gaze direction, and the determined position of the vehicle operator's hand.

[0012] In another aspect, it describes that the vehicle component may be a steering component of a vehicle and the method may include actuating the steering component when the position of the hand is not in contact with a steering element of the vehicle and the instantaneous direction is away from a forward direction with respect to the vehicle.

[0013] In another aspect, it describes that the vehicle component may be a human-machine interface (HMI) of a vehicle and the method may include activating the HMI when the passenger is present, the position of the hand is in contact with a steering element of the vehicle, and the eye direction is in a forward direction with respect to the vehicle.

[0014] In another aspect, it describes that the method may include deactivating the HMI when the position of the hand is not in contact with the steering element of the vehicle or the eye direction is not in the forward direction.

[0015] In another aspect, it describes that determining a position of a hand of the vehicle operator may include using the image of the vehicle operator to create a node model of the vehicle operator.

[0016] The embodiments, examples, and alternatives of the preceding paragraphs, the claims, or the following description and drawings, including any of their various aspects or respective individual features, may be considered independently of one another or in any combination. Features described in connection with one embodiment are applicable to all embodiments, unless such features are inconsistent. BRIEF DESCRIPTION OF THE DRAWINGS Fig. 1 is a block diagram of an example vehicle. Fig. 2A is an example camera view of a driver seated in a vehicle. Fig. 2B is an example node model of the driver from Fig. 2A. Fig. 3 is an exemplary vehicle interior within a driver's field of vision. Fig. 4 is a process flow diagram illustrating an example process for driver intervention-based vehicle component actuation. DETAILED DESCRIPTION

[0017] This disclosure provides techniques for controlling vehicle components, features, and / or systems in response to determining whether a vehicle operator's hands are in contact with a vehicle steering wheel and the vehicle operator's gaze is in a forward direction with respect to the vehicle, e.g., toward a road. In one example, a touchscreen display associated with, e.g., one or more of an infotainment system, a vehicle navigation system, and / or a communication system may be activated to interact with a vehicle passenger in response to determining that the vehicle operator's hands are in contact with the steering wheel and the vehicle operator's gaze is on the road. In some examples, activation of the touchscreen is further dependent on the presence of a passenger.In one example, other or additional components of the vehicle may be actuated in addition to or instead of the touchscreen display. For example, a steering component may be actuated when the vehicle operator's hands are not in contact with the steering wheel and / or the vehicle operator's line of sight is away from the road. In examples, a camera mounted on an instrument panel of the vehicle and / or other sensors may acquire images that may include a vehicle operator positioned in the driver's seat of the vehicle. These images may be used to determine a vehicle operator's gaze direction and positions of the vehicle operator's hands. Certain portions of the vehicle operator, such as one or more of the vehicle operator's hands, forearms, etc., may not be visible within the camera's field of view.A computer can execute programming that uses the images to create a nodal model of the driver. The computer can additionally estimate positions of features of the driver that are obscured from the camera's field of view. The computer can estimate the driver's gaze direction based on the images. Gaze direction can be estimated, for example, using eye-tracking algorithms. Gaze direction can then be mapped to zones associated with the windshield and dashboard to indicate whether the driver is looking at the road or another location within the vehicle's interior, such as the infotainment system, vehicle navigation system, and / or communications system.

[0018] This document discloses a system including a computer having a processor and a memory. The memory includes instructions executable by the processor for determining a presence of a passenger, determining a gaze direction of a vehicle operator based on an image of the vehicle operator, and determining a position of a hand of the vehicle operator based on the image of the vehicle operator. The system actuates a vehicle component based on the determined presence of the passenger, the determined gaze direction of the vehicle operator, and the determined position of the vehicle operator's hand.

[0019] The vehicle component may be a steering component of a vehicle, and the instructions may include instructions for operating the steering component when the position of the hand is not in contact with a steering element of the vehicle and the instantaneous direction is away from a forward direction with respect to the vehicle.

[0020] The vehicle component may be a human-machine interface (HMI) of a vehicle, and the instructions may include instructions for activating the HMI when the passenger is present, the position of the hand is in contact with a steering element of the vehicle, and the eye direction is in a forward direction with respect to the vehicle.

[0021] The instructions may include instructions to deactivate the MMS if the position of the hand is not in contact with the steering element of the vehicle or the direction of the eye is not in the forward direction.

[0022] The instructions for determining a gaze direction may include instructions for using the image of the vehicle operator to determine a gaze angle and map the gaze angle to a direction relative to a vehicle.

[0023] The instructions for determining a position of a hand of the vehicle operator may include instructions for using the image of the vehicle operator to create a node model of the vehicle operator.

[0024] The instructions may include instructions for determining a gaze direction of the passenger based on an image of the passenger and determining a position of a hand of the passenger based on the image of the passenger.

[0025] The vehicle component may be a vehicle's HMI, and the instructions may include instructions for activating the HMI when the passenger is present, the position of the vehicle operator's hand is in contact with a steering element of the vehicle, the vehicle operator's eye-direction is in a forward direction relative to the vehicle, the passenger's eye-direction is toward the HMI, and the position of the passenger's hand is in contact with the HMI. The instructions may include instructions for deactivating the HMI when the passenger is not present. The HMI may be a vehicle navigation system.

[0026] Disclosed herein is a method that includes determining a passenger's presence, determining a vehicle operator's gaze direction based on an image of the vehicle operator, and determining a position of a vehicle operator's hand based on the image of the vehicle operator. The method may include actuating a vehicle component based on the determined passenger's presence, the determined vehicle operator's gaze direction, and the determined position of the vehicle operator's hand.

[0027] The vehicle component may be a steering component of a vehicle and the method may include actuating the steering component when the position of the hand is not in contact with a steering element of the vehicle and the instantaneous direction is away from a forward direction with respect to the vehicle.

[0028] The vehicle component may be a human-machine interface (HMI) of a vehicle, and the method may include activating the HMI when the passenger is present, the position of the hand is in contact with a steering element of the vehicle, and the eye direction is in a forward direction with respect to the vehicle.

[0029] The method may include deactivating the HMI when the position of the hand is not in contact with the steering element of the vehicle or the eye direction is not in the forward direction.

[0030] Determining a gaze direction may include using the image of the vehicle driver to determine a view angle and map the view angle to a direction relative to a vehicle.

[0031] Determining a position of a hand of the vehicle operator may include using the image of the vehicle operator to create a node model of the vehicle operator.

[0032] The method may include determining a gaze direction of the passenger based on an image of the passenger and determining a position of a hand of the passenger based on the image of the passenger.

[0033] The vehicle component may be a vehicle's HMI, and the method may include activating the HMI when the passenger is present, the position of the vehicle operator's hand is in contact with a steering element of the vehicle, the vehicle operator's eye-direction is in a forward direction relative to the vehicle, the passenger's eye-direction is toward the HMI, and the position of the passenger's hand is in contact with the HMI. The method may include deactivating the HMI when the passenger is not present. The HMI may be a vehicle navigation system.

[0034] Fig. 1 is a block diagram of an exemplary vehicle. As shown in Fig. 1, the system 100 includes the vehicle 102, which includes the computer 104 communicatively coupled to various elements, including the sensors 108, subsystems or components 110 such as steering, propulsion, braking, the human-machine interface (HMI) 112, and the communication component 114, via the vehicle network 106. The computer 104 and the server 118, discussed below, include a processor and a memory. Memory of the computer 104, such as those described herein, includes one or more forms of non-transitory media readable by the computer 104 and can store instructions executable by the computer 104 for performing various operations, such that the vehicle computer is configured to perform the various operations, including those disclosed herein.

[0035] For example, computer 104 may include a generic computer with a processor and memory, as described above, and / or may include an electronic control unit (ECU) or controller for a specific function or set of functions and / or a dedicated electronic circuit, including an ASIC (application specific integrated circuit) fabricated for a specific operation (e.g., an ASIC for processing data from sensors and / or communicating data from sensors 108). In another example, computer 104 may include an FPGA (field programmable gate array), which is an integrated circuit fabricated to be configurable by a user.In examples, a hardware description language such as VHDL (Very High-Speed Integrated Circuit Hardware Description Language) may be used in electronic design automation to describe digital and mixed-signal systems, such as FPGAs and ASICs. For example, an ASIC is manufactured based on VHDL programming provided prior to manufacture, whereas logical components within an FPGA may be configured based on the VHDL programming, e.g., stored on memory electrically connected or coupled to the FPGA circuitry. In some examples, a combination of processor(s), ASIC(s), and / or FPGA circuitry may be included in computer 104. Further, computer 104 may include a plurality of computers in the vehicle (e.g.,a plurality of ECUs or the like) that operate together to perform operations attributed to the computer 104 in this document.

[0036] Memory of computer 104 may include any type, such as hard disk drives, solid-state drives, or any volatile or non-volatile media. The memory may store the collected data transmitted by sensors 108. The memory may be a separate device from computer 104, and computer 104 may retrieve information stored by the memory via a communication network in the vehicle, such as vehicle network 106, e.g., via a controller area network (CAN) bus, a local interconnect network (LIN) bus, a wireless network, etc. Alternatively or additionally, the memory may be part of computer 104, for example, as internal memory of computer 104.

[0037] The computer 104 may include or access instructions to operate one or more components 110 such as vehicle brakes, propulsion (e.g., one or more of an internal combustion engine, an electric motor, a hybrid engine, etc.), steering, climate control, interior and / or exterior lights, infotainment, navigation, etc., as well as to determine if and when the computer 104, as opposed to a human vehicle operator, should control such operations. The computer 104 may include or be communicatively coupled to more than one processor, for example, via the vehicle network 106, which may be included in, for example, components 110 such as sensors 108, ECUs, or the like included in the vehicle for monitoring and / or controlling various vehicle components, e.g., a powertrain controller, a braking controller, a steering controller, etc.

[0038] The computer 104 may generally be arranged for communication in the vehicle network 106, which may include a communication bus in the vehicle, such as a Controller Area Network (CAN) or the like, and / or other wired and / or wireless mechanisms. The vehicle network 106 corresponds to a communication network that may facilitate the exchange of messages between various on-board vehicle devices, e.g., sensors 108, components 110, the computer 104, and a computer onboard the vehicle 102. The computer 104 may generally be programmed to send and / or receive messages to and / or from other devices in the vehicle, e.g., any or all of ECUs, sensors 108, actuators, components 110, a communication module, an HMI 112, via the vehicle network 106. For example, various subsystems of components 110 (e.g., components 110) may be controlled by respective ECUs.

[0039] Furthermore, in implementations where computer 104 actually includes a plurality of devices, vehicle network 106 may be used for communication between devices depicted in this disclosure as computer 104. For example, vehicle network 106 may provide communication capability over a wired bus, such as a CAN bus, a LIN bus, or utilize any type of wireless communication capability. Vehicle network 106 may include a network in which messages are communicated using any other wired communication technologies and / or wireless communication technologies, e.g., Ethernet, Wi-Fi®, Bluetooth®, etc. Additional examples of protocols that may be used in some implementations to communicate over vehicle network 106 include, but are not limited to, Media Oriented System Transport (MOST), Time-Triggered Protocol (TTP), and FlexRay.In some implementations, the vehicle network 106 may represent a combination of multiple networks, possibly of different types, that support communication between devices onboard a vehicle. For example, the vehicle network 106 may include a CAN bus, in which some in-vehicle sensors and / or components communicate via a CAN bus, and a wired or wireless local area network, in which some devices in the vehicle communicate according to Ethernet, Wi-Fi®, and / or Bluetooth communication protocols.

[0040] The vehicle 102 typically includes a variety of sensors 108, including torque sensors, capacitive sensors, and other sensors related to determining whether a vehicle operator has placed a hand in contact with the steering wheel 124. The sensors 108 may include any number of devices capable of obtaining one or more measurements of one or more physical phenomena. Some of the sensors 108 detect variables that characterize the operating environment of the vehicle, such as vehicle speed settings, vehicle towing parameters, vehicle braking parameters, engine torque output, engine and transmission temperatures, battery temperatures, vehicle steering parameters, etc. Some of the sensors 108 detect variables that characterize the physical environment of the vehicle 102, such as ambient air temperature, humidity, weather conditions (e.g., rain, snow, etc.).), parameters related to the slope or gradient of a road or other type of path on which the vehicle is traveling, etc. In examples, the sensors 108 may be operable to detect the position or orientation of the vehicle, for example, using signals from a satellite positioning system (e.g., the Global Positioning System or GPS); accelerometers, such as piezoelectric or microelectromechanical systems (MEMS); gyroscopes, such as rate, ring laser, or fiber optic gyrometers; inertial measurement units (IMU); and magnetometers. In examples, the sensors 108 may include sensors for detecting aspects of the environment external to the vehicle 102, such as radar sensors, scanning laser rangefinders, cameras, etc.Sensors 108 may also include light detection and ranging (LIDAR) sensors (LIDAR sensors), which operate to detect distances to objects by emitting a laser pulse and measuring the time of travel of the pulse to the object and back. Sensors 108 may include a controller and / or microprocessor executing instructions, for example, to perform analog-to-digital conversion to convert sampled analog measurements and / or observations into input signals that may be provided to computer 104, e.g., via vehicle network 106.

[0041] Sensors 108 may include occupancy sensors 122 to identify whether an occupant is sitting in one or more of the seats. Occupancy sensor 122 may be, for example, a weight sensor, an image detector, a buckled seatbelt, etc. Vehicle 102 may include any suitable number of occupancy sensors 122.

[0042] For example, the vehicle 102 may include a number of occupancy sensors 122 equal to the number of seats in the vehicle 102, including, for example, vehicle operator and front passenger seats. In some examples, the occupancy sensors 122 may be of a conventional type currently known in the art.

[0043] The computer 104 may be configured to utilize vehicle-to-vehicle (V2V) communication via the communication component 114 and / or may interface with devices external to the vehicle, e.g., through a wide area network (WAN) 116 via V2V communication. The computer 104 may communicate external to the vehicle 102, such as via vehicle-to-infrastructure (V2I) communication, vehicle-to-everything (V2X) communication, or V2X including cellular communications (C-V2X) and / or wireless communications, cellular with dedicated short range communications (DSRC), etc. Communication external to the vehicle 102 may be facilitated through direct radio frequency communication and / or via the network server 118.The communication component 114 may include one or more mechanisms by which the computer 104 communicates with vehicles external to the vehicle 102, including any desired combination of wireless, e.g., cellular, wireless, satellite, microwave, and radio frequency communication mechanisms, and any desired network topology or topologies when a plurality of communication mechanisms are utilized.

[0044] The vehicle 102 may include the MMS 112, e.g., one or more of an infotainment display, a touchscreen display, a microphone, a speaker, a haptic device, etc. A user, such as the vehicle driver and / or a passenger of the vehicle 102, may provide input to devices, such as the computer 104, via the MMS 112. The MMS 112 may communicate with the computer 104 via the vehicle network 106; e.g., the MMS 112 may send a message including user input provided via a touchscreen, a microphone, a camera capturing a gesture, etc., to the computer 104, and / or may display output, e.g., via a display, a speaker, etc. Further, the operations of the MMS 112 may be performed by a portable user device (not shown), such as a smartphone or the like, that is in communication with the computer 104, e.g., a mobile phone. E.g. via Bluetooth or similar.

[0045] The WAN 116 may include one or more mechanisms by which the computer 104 can communicate with the server 118. The server 118 may include a device having one or more computing devices, e.g., having respective processors and memory and / or associated data stores accessible via the WAN 116. In examples, the vehicle 102 may include a wireless transceiver (i.e., a transmitter and / or receiver) to send and receive messages external to the vehicle 102. Accordingly, the network may include one or more of various wired or wireless communication mechanisms, including any desired combination of wired communication mechanisms, e.g., cable and fiber communication mechanisms, and / or wireless communication mechanisms, e.g.,Cellular, wireless, satellite, microwave, and radio frequency communication mechanisms, and any desired network topology or network topologies when multiple communication mechanisms are utilized. Example communication networks include wireless communication networks, e.g., using Bluetooth, Bluetooth Low Energy (BLE), IEEE 802.11, V2V, or V2X, such as cellular V2X (CV2X), DSRC, etc., local area networks, and / or wide area networks 116, including the Internet.

[0046] In one example, the computer 104 may capture an image of a portion of an interior of the vehicle 102, which may include a vehicle operator seated in the driver's seat of the vehicle 102, using a camera 210 mounted on the dashboard of the vehicle 102 ( Fig. 2A). In some examples, the image of the vehicle operator 205 includes, for example, the head, neck, shoulders, chest, and / or upper arms of the vehicle operator, but may exclude an image of the vehicle operator's hands. Accordingly, suitable image processing programming executed by the computer 104 may extract parameters of the image to create a node model (described in more detail below with respect to Fig. 2B) of the portion of the vehicle operator's body that lies within the camera's field of view 215. Computer 104 may additionally execute programming to estimate the positions of features that are excluded or obscured from the camera's field of view.

[0047] The computer 104 may further execute programming to determine a level of confidence regarding whether the nodal model of the vehicle operator matches a nodal model of a vehicle operator having one or more hands placed in contact with the steering wheel 124. In response to the confidence level meeting a predetermined threshold (e.g., greater than 95%) that the vehicle operator's hand(s) are at a position on the steering wheel, the computer 104 may execute programming to actuate or generate a signal to indicate that one or more of the vehicle operator's hands are in contact with the steering wheel 124.Conversely, in response to the confidence level being below the predetermined threshold, the computer 104 may execute programming to actuate or generate a signal to indicate that one or more of the vehicle operator's hands are not in contact with the steering wheel 124. It should be noted that, although . Fig. 1 shows a steering wheel 124, techniques described herein may be applied to steering components other than a vehicle steering wheel, such as a joystick, an aircraft control horn, etc.

[0048] A computer may include programming to determine the presence of a passenger, i.e., a person other than a vehicle operator or driver, and who is therefore sitting in a position other than an operator or driver position. If a passenger is present, the computer then determines whether the vehicle operator is fully engaged in the operation of the vehicle, i.e., has their hands on the steering wheel and their eyes on the road. The computer may determine an eye-gaze direction of the vehicle operator and positions of the vehicle operator's hands based on an image of the vehicle operator. The computer may then actuate a vehicle component (e.g., vehicle HMI) based on the determined presence of the passenger, the determined eye-gaze direction of the vehicle operator, and the determined positions of the vehicle operator's hands.

[0049] Fig. 2A is an exemplary camera view 200 of the driver 205 sitting in the vehicle 102. The camera view from Fig. 2A may be obtained via the dashboard-mounted camera 210 of the vehicle 102, which may be positioned near a central portion of the dashboard of the vehicle 102. The dashboard-mounted camera 210 may provide a field of view 215 that includes the vehicle operator 205 in a driving position of the vehicle and a passenger to the right of the vehicle operator 205, as well as passengers located behind the vehicle operator 205. The dashboard-mounted camera 210 may include any camera imaging device capable of detecting electromagnetic radiation in a range of wavelengths. For example, the dashboard-mounted camera 210 may detect visible light, infrared radiation, ultraviolet light, or some range of wavelengths including visible, infrared, and / or ultraviolet light.For example, a dashboard-mounted camera may include image sensors such as charge-coupled devices (CCDs), active-pixel sensors such as complementary metal-oxide semiconductor (CMOS) sensors, etc. In some examples, images from the dashboard-mounted camera 210 are supplemented or augmented by additional non-camera sensors of the sensors 108. In some examples, images captured by the dashboard-mounted camera 210 may be supplemented and / or fused with images captured by other cameras of the vehicle 102.Alternatively or additionally, images of the vehicle operator 205, which may be fused with or supplemented by images from other cameras of the vehicle 102, may be fused with depth measurements and / or speed measurements indicative of movements of the vehicle operator 205 based on signals from a radar mounted on the dashboard.

[0050] As in Fig. 2A, the vehicle operator 205 appears to have a gaze direction 220 away from a forward direction 150 with respect to the vehicle 102 in order to view and / or interact with a vehicle HMI 112, e.g., an infotainment display, a vehicle navigation system, and / or a communication system, etc. In this context, a "forward direction" means that the vehicle operator's gaze is considered to be directed in a direction consistent with an intervention in operating the vehicle. A forward direction could be defined as an area or space with respect to a vehicle coordinate system; for example, a longitudinal axis of a vehicle could be specified in a coordinate system, and then the forward direction could be defined by a cone described by a line at a specified angle to the longitudinal axis.Alternatively or additionally, the vehicle driver's gaze could be directed in a forward direction 150 when looking through the windshield at the road ahead. Even further alternatively or additionally, a forward direction 150 could be defined by a portion of a rearview mirror, a side view mirror, an instrument cluster, etc. Further, the forward direction could be defined dynamically, i.e., it could change based on vehicle circumstances and / or current events, or be different at different times. For example, a forward direction 150 could be defined as passing through a windshield of the vehicle unless a steering angle of the vehicle exceeds a threshold, at which point a forward direction could be defined as passing through or including a vehicle side window, e.g., when turning a corner.The driver's gaze being "off" the forward direction means that the driver's gaze deviates from the forward direction by more than a predetermined angle and / or does not fall within a space or area (e.g., a cone or other field of view, an area defined by a windshield) that defined the forward direction. A non-forward direction or direction off the forward direction further typically results in the driver's gaze direction being considered inconsistent with interfering with the operation of the vehicle, such as looking at the MMS 112 of the vehicle 102, e.g., infotainment display, a vehicle navigation system, and / or a communication system, etc.

[0051] As also in Fig. 2A, the vehicle operator 205 has lowered his right shoulder in a manner that may be consistent with interacting with a control element mounted on the instrument panel of the vehicle 102. As further shown in Fig. 2A, the hand positions of the vehicle operator 205 are obscured from the view of the dashboard-mounted camera 210. However, detection of whether the vehicle operator 205 has placed one or more hands in contact with the steering wheel 124 of the vehicle 102 may be estimated through additional processing, such as image processing and other programming of the computer 104, as described in more detail below.

[0052] Fig. 2B is an exemplary node model 250 of the vehicle driver 205 from Fig. 2A. In this context, a "node model" means a representation of a vehicle driver 205 by a system of nodes connected by lines, such that the node model 250 can represent body parts or features of the vehicle driver 205. Thus, in the example from Fig. 2B an image of the driver 205 through nodes (solid circles in Fig. 2B) representing the shoulders, elbows, hips, etc. of the vehicle operator 205, with lines drawn to connect the nodes to represent such features. The node model 250 may be generated using a suitable skeletonization technique operated to reduce captured images, for example, to representations having a width of one pixel. In this context, a "skeletonization" technique means representing features of the vehicle operator 205 in the image by a node model just described. Thus, in Fig. 2B, a skeletonization technique is applied to the vehicle driver 205 to represent features of the vehicle driver 205 through the node model 250. In one example, the image processing programming of the computer 104 may skeletonize an image of the vehicle driver 205 by creating successive passes of the image acquired by the dashboard-mounted camera 210. According to a skeletonization algorithm, during successive frame-by-frame processing operations, edge pixels may be identified and removed from the current image frame based on whether the edge pixels of a current image frame break connectivity with an image frame represented by the remaining pixels of the image in a subsequent frame.Accordingly, a node model 250 may be generated based on the image processing programming determining that all or substantially all pixels remaining in a current frame maintain their connectivity with an image from a previous frame.

[0053] In response to generating the node model 250, the computer 104 may execute programming to determine whether the node model is consistent with a node model of a vehicle operator having one or more hands in contact with, e.g., resting on, the steering wheel 124. For example, the computer 104 may utilize a measure of an angle of a line drawn between nodes representing the shoulders of the vehicle operator 205. Alternatively or additionally, the computer 104 may utilize a measure of an angle between a line drawn between nodes representing the shoulders of the vehicle operator 205 and a line representing an upper arm of the vehicle operator. Alternatively or additionally, the computer 104 may utilize a measure of an angle between a line representing an upper arm of the vehicle operator 205 and the forearm of the vehicle operator.Alternatively or additionally, the computer 104 may use a distance between a line representing an upper arm of the vehicle operator 205 and a point on a line drawn between nodes representing the shoulders of the vehicle operator 205. The computer 104 may use additional parameters of the node model 250 in determining whether the node model is consistent with a node model of a vehicle operator having one or more hands in contact with the steering wheel 124.

[0054] In one example, programming of the computer 104 to determine whether the generated node model 250 is consistent with the node model of a vehicle driver having a hand or hands in contact with the steering wheel may include a machine learning technique, e.g., a neural network, capable of predicting, based on the node model 250, whether one or more hands of the vehicle driver 205 are in contact with the steering wheel 124, e.g., resting thereon. For example, a neural network could be trained to receive an image of the vehicle driver 205 aggregated with additional data, such as data from other cameras / sensors in the interior of the vehicle 102 describing the position and movements of the vehicle driver. The neural network could predict the skeletonization model, confidence, and hands-on-wheel status. The predicted confidence of the skeletonization model, e.g., E.g. hands-on status, etc., can be compared to actual instances where the vehicle operator 205 has one or more hands in contact with the steering wheel 124. In one example, data may be collected and referred to as ground truth data, which is used to train the neural network and / or to train another machine learning technique. In response to the initial training of a machine learning implementation, cross-correlation and / or time series analyses could be used to determine whether a machine learning implementation is consistent across a window of time, such as a window of time in which the vehicle operator 205 temporarily places their hands in contact with the steering wheel 124.

[0055] In some examples, programming of computer 104 performs a comparison of node model 250 with node models of a vehicle operator known to have one or more hands placed in contact with steering wheel 124. Such reference node models or parameters derived from node models may be stored in memory accessible by computer 104. In such cases, node models of a vehicle operator with one or more hands in contact with steering wheel 124 may be compared with node models of a vehicle operator without one or more hands spaced from the steering wheel.For example, a node model of a vehicle operator with a hand placed in contact with the steering wheel 124 may show a specific angle or a specific range of angles between a line representing an upper arm and a line drawn between nodes representing the vehicle operator's shoulders. Thus, in response to the node model 250 showing a similar angle between a line representing an upper arm of the vehicle operator 205 and a line drawn between nodes representing the vehicle operator's 205 shoulders, the computer 104 may determine a level of confidence that at least one hand of the vehicle operator 205 is not in contact with the steering wheel 124.

[0056] In one example, computer 104 may execute programming to estimate a position or positions of one or more hands or other features of vehicle operator 205 based on comparisons to node models, e.g., node models derived from images of vehicle operators similar to vehicle operator 205 (e.g., approximately the same torso and / or arm length), stored in a memory of computer 104. In one example, computer 104 may define a coordinate system, e.g., a three-dimensional (3D) Cartesian coordinate system, with a specified origin and orthogonal X, Y, and Z axes for the interior of vehicle 102. The coordinate system may be used to describe attributes of camera 210, such as camera pose, focal length, etc.Further, defining a coordinate system for the interior of the vehicle 102 may allow the computer 104 to specify positions of the nodes of the node model 250 relative to the defined coordinate system. For example, the computer 104 may specify positions of the nodes of the node model 250 relative to the defined coordinate system in response to determining that a line representing an upper arm portion of the vehicle operator 205 slopes in a direction toward the base of the driver's seat, as shown in FIG. Fig. 2B, estimate the position of the right forearm and hand of the vehicle operator 205 based on comparisons with node models stored in a memory of the computer 104. In another example, also shown in Fig. As can be seen in Figure 2A, the node representing the left elbow of the vehicle operator 205 is located below the node representing the vehicle operator's left shoulder. Accordingly, the computer 104 can estimate the position of the left forearm and left hand of the vehicle operator 205 based on comparisons with node models stored in a memory of the computer 104.

[0057] In some examples, the programming of computer 104 may implement a suitable learning model of the vehicle operator 205 that refines estimates of whether the hands of the vehicle operator 205 are in contact with the steering wheel 124 over time. For example, a learning model may include using a node model that includes a node representing the head position relative to other nodes of the vehicle operator 205, a gaze direction of the vehicle operator, other positions of nodes representing the arms of the vehicle operator 205, and so on.

[0058] In one example, node models of vehicle drivers known to have one or more hands placed in contact with the steering wheel 124 (which may be referred to as reference models) may be stored in memory accessible by the computer 104. A determination of whether the node model 250 is consistent with a reference node model of a vehicle driver known to have one or more hands placed in contact with the steering wheel 124 may be the result of applying a suitable supervised machine learning process, such as via the server 118, which may utilize a corpus of node models representing vehicle drivers having one or more hands in contact with the steering wheel 124 or (conversely) having one or more hands separated therefrom.In another example, a suitable unsupervised machine learning technique may be utilized, such as a generative adversarial network that performs unsupervised development and / or refinement of a process for determining whether one or more hands of the vehicle operator 205 are in contact with the steering wheel 124. Thus, for example, during an unsupervised training process, programming steps performed by the computer 104 and / or the server 118 may generate node models of vehicle operators whose hands are placed in contact with the steering wheel 124 (e.g., "hands on") or, conversely, are separated from it (e.g., "hands off"). Such node models may be evaluated and / or verified by programming the computer 104 without the benefit of obtaining an advanced indication of the hands-on / hands-off state of the generated node models.In response to computer 104 identifying a hands-on / hands-off state of vehicle operator 205, one or more parameters of a generated node model may be modified. In response to a modification of parameters of a node model, the generative adversarial network may present the same or a similar node model for review by computer 104. Accordingly, in an exemplary training environment, which may be hosted by server 118 cooperating with computer 104, the generative adversarial network may iteratively repeat a process of generating a node model, followed by attempting hands-on / hands-off detection by programming computer 104, refining parameters of a node model, generating another node model, etc.

[0059] The computer 104 may then execute programming to determine a level of confidence that estimated feature positions of a vehicle operator, e.g., the estimated feature positions 255, 260, match feature positions of a vehicle operator with a hand (or hands) placed in contact with the steering wheel 124. In this context, a "level of confidence" or "confidence level" means a degree to which the node model 250 matches a node model known to represent a vehicle operator having one or more hands placed in contact with the steering wheel 124.Thus, for example, a perfect match between the node model 250 and a node model stored in memory accessible by the computer 104 of a vehicle driver known to have one or more hands in contact with the steering wheel 124 may be assigned a confidence level of 100%. In another example, a significant mismatch between the node model 250 and a node model stored on or accessible by the computer 104 may be assigned a lower confidence level, such as 80%, 75%, etc.

[0060] Furthermore, in this context, an "estimated feature position" means a predicted position of a limb of the vehicle operator 205 that is excluded or occluded from the view of the camera 210, based on observed positions of the features of the vehicle operator 205 represented by the node model 250. For example, as in Fig. 2B, the estimated feature position 255, which corresponds to the right forearm and right hand of the vehicle operator 205, is near the estimated feature position 270, which represents the legs of the vehicle operator 205. Thus, programming of the computer 104 may assign a relatively low level of confidence that the right hand of the vehicle operator 205 is in contact with the steering wheel 124. In another example, also shown in Fig. 2B, the estimated feature location 260 corresponding to the left forearm and hand of the vehicle operator 205 is positioned, for example, in the coordinate system of the interior of the vehicle 102 at a position approximating the steering wheel 124. Thus, the computer 104 can assign a relatively high level of confidence that the right hand of the vehicle operator 205 is in contact with the steering wheel 124.

[0061] In one example of a machine learning model implementing a neural network, a confidence level may be increased using a Bayesian neural network or an ensemble approach, where the variance of a confidence level may be reduced by generating additional data for training from a dataset using combinations with repetitions to generate multiple sets of an original dataset. Further, in one example, a confidence level may be increased using input signals from other components of the HMI 112 of the vehicle 102. For example, in response to the vehicle operator 205 interacting with an infotainment component of the vehicle 102, which may be validated using input signals from the infotainment component, may also be validated to increase predictions or confidence levels of the nodal model 250.In some examples, inputs from torque sensors and / or capacitive sensors may be used to modify the confidence level and / or support machine learning techniques. In another example, a first node model could be generated using features extracted from images captured by camera 210 and a second camera having (typically overlapping, but not identical) fields of view including the interior of vehicle 102. Node models resulting from programming computer 104 that extract features captured by camera 210 and the second camera could improve the confidence in predicting the position of the vehicle operator's hands 205.

[0062] In one example, the gaze direction 220 may be determined using eye tracking, gaze estimation, facial recognition, and / or other suitable algorithms, including those conventionally known. The gaze direction may be characterized according to a line or ray that may be determined to have an angle with respect to a forward direction 150 and / or that may be determined to be within a forward direction 150, e.g., when mapped to zones on the windshield and / or instrument panel using, for example, a lookup table. The zones, e.g., 302, 304, and 306, may correspond to a direction with respect to the vehicle 102. As in Fig. 3, the vehicle driver's gaze direction 220 may be mapped to different zones, including forward directions, such as the windshield 302 and the instrument cluster 306, as well as non-forward directions, such as the MMS zone 304. In the Fig. 2A, the vehicle driver 205 appears to have a gaze direction 220 away from the forward direction 150 to view the MMS 112 in zone 304. The MMS zone 304 may include the MMS 112 for an infotainment display, a vehicle navigation system, and / or a communication system, etc. The MMS may include a touchscreen display 308 and / or buttons 310, etc. In some examples, the vehicle driver's gaze direction and / or a corresponding zone may be provided by a vehicle driver state monitoring (DSM) system.

[0063] Although the node model and gaze direction are shown and described with respect to an operator of the vehicle, these techniques can also be applied to the front seat passenger of the vehicle. The passenger can be monitored to determine, for example, whether the passenger's gaze direction is toward the MMS and whether the position of the passenger's hand is in contact with the MMS. In some examples, the node model can be used to distinguish between the operator's hands and the passenger's hands, for example, based on size. In addition, the hands can be distinguished based on other characteristics, such as hairiness, skin color, jewelry, freckles, etc.

[0064] Fig.4 is a process flow diagram 400 illustrating an example process for controlling vehicle components, features, and / or systems in response to determining whether a vehicle operator's hands are in contact with a vehicle steering wheel 124 and the vehicle operator's gaze is in a forward direction with respect to the vehicle 102, e.g., toward the road. The process 400 may be implemented on a computer 104 included in a vehicle 102. The process 400 includes a plurality of blocks that may be executed in the illustrated order. The process 400 could alternatively or additionally include fewer blocks or include the blocks being executed in different orders.

[0065] The process 400 may begin at decision block 402, such as in response to the vehicle 102 being placed in an on or "drive" state, for example, to operate on a roadway. At block 402, the computer 104 may determine whether the vehicle 102 is moving. For example, the computer may monitor a speed of the vehicle 102. In response to a determination at decision block 402 that the vehicle is moving, e.g., the vehicle speed is greater than zero, the process 400 may proceed to block 404. Otherwise, the process 400 may return to decision block 402 to monitor whether the vehicle is moving.

[0066] At block 404, the computer 104 may, for example, deactivate an MMS for an infotainment display, a vehicle navigation system, and / or a communication system. Alternatively, the system may deactivate specific features of the MMS, including a written input, such as an address or a search string.

[0067] Process 400 may continue at block 406, which may include computer 104 monitoring for the presence of a passenger. Specifically, computer 104 may monitor passenger-side occupancy sensor 122 for the presence of a front-seat passenger.

[0068] The process 400 may continue at block 408, which may include the computer 104 monitoring a gaze direction of the vehicle operator. The computer 104 may determine the gaze direction 220 using facial recognition or other suitable algorithms. The gaze direction may be characterized as a view angle (e.g., between the forward direction 150 and a ray corresponding to the gaze direction 220), which may be mapped to zones on the windshield and / or instrument panel. The vehicle operator's gaze direction 220 may be mapped to zones that include forward directions and non-forward directions, such as the MMS zone 304.

[0069] The process 400 may continue at block 410, which may include the computer 104 monitoring hand positions of the vehicle operator. Block 410 may include the dashboard-mounted camera 210 capturing an image of an interior of the vehicle 102, which may include the vehicle operator 205, where the captured image may exclude certain features, such as the forearms and hands of the vehicle operator 205. The image is then processed to form a node model and estimate positions of features excluded from the camera's field of view. For example, the computer 104 may estimate positions of the vehicle operator's left and right hands to determine whether or not the vehicle operator's hands are on the steering wheel.

[0070] The process 400 may continue at decision block 412, which may include the computer 104 determining whether a passenger is present. In response to a determination at decision block 412 that a passenger is present, the process 400 may proceed to decision block 414. Otherwise, the process 400 returns to block 404, where the MMS is deactivated or remains deactivated, and the system continues to monitor the status of the passenger and the vehicle operator, i.e., as described with respect to blocks 406-416. In one example, in response to a determination at decision block 412 that no passenger is present, the computer 104 may actuate or generate a message to the vehicle operator 205 via the MMS 112 notifying the vehicle operator that the deactivated MMS or a deactivated feature of the MMS is only available when a passenger is present.

[0071] At decision block 414, the computer 104 may determine whether the vehicle operator's instantaneous direction is in a forward direction. In response to a determination at decision block 414 that the vehicle operator's instantaneous direction is in a forward direction, the process 400 may proceed to decision block 416. Otherwise, the process 400 returns to block 404, where the HMI 112 is deactivated or remains deactivated, and the system continues to monitor the status of the passenger and the vehicle operator, i.e., blocks 406-416. In an alternative example, the computer 104 may determine whether the vehicle operator's instantaneous direction is not in the direction of the HMI 112. In response to a determination that the vehicle operator's instantaneous direction is not in the direction of the HMI 112, the process 400 may proceed to decision block 416.

[0072] At decision block 416, the computer 104 may determine whether one or more of the vehicle operator's hands are in contact with the steering wheel. In one example, the computer may determine that both of the vehicle operator's hands are on the steering wheel or that at least the hand closest to the HMI is in contact with the steering wheel. In response to a determination at decision block 416 that at least the hand closest to the HMI is in contact with the steering wheel, the process 400 may proceed to block 418. Otherwise, the process 400 returns to block 404, where the HMI is deactivated or remains deactivated, and the system continues to monitor the status of the passenger and the vehicle operator, i.e., blocks 406-416. In an alternative example, the computer 104 may determine whether the vehicle operator's hands are not in contact with the HMI.In response to a determination that the vehicle operator's hands are not in contact with the MMS, process 400 may proceed to block 418.

[0073] If, at block 418, the computer 104 has determined that a passenger is present (block 412), the eye direction is toward or in a forward direction (block 414), and that at least the hand closest to the HMI is in contact with the steering wheel (block 416), the computer 104 may activate the HMI. After block 418, the system continues to monitor the status of the passenger and the vehicle operator, i.e., blocks 406-416.

[0074] In one example, process 400 may include monitoring a gaze direction of the passenger based on an image of the passenger and monitoring the positions of the passenger's hands based on the image of the passenger. In addition to requiring the passenger to be present to activate the HMS, the system may additionally require that the passenger's gaze direction be toward the HMS, e.g., an axis of the gaze direction intersects a face of the HMS, and that the position of at least one of the passenger's hands be in contact with the HMS.Thus, in one example, the system activates the MMS only in response to determining that the passenger is present, the position of the vehicle operator's hand is in contact with a steering element of the vehicle, the vehicle operator's instantaneous direction is toward or in a forward direction with respect to the vehicle, the passenger's instantaneous direction is toward the MMS, and the position of the passenger's hand is in contact with the MMS.

[0075] In an alternative example, the computer 104 may issue a command to control a subsystem or component 110, including steering, braking, and / or propulsion of the vehicle 102, for example, to steer and / or decelerate the vehicle 102 in response to determining that a vehicle operator's hands are not in contact with the steering wheel and / or the gaze direction is away from a forward direction. For example, the computer 104 may actuate a steering component when the hand position is not in contact with a steering element of the vehicle and the gaze direction is away from a forward direction relative to the vehicle.

[0076] Operations, systems and procedures described in this document should always be implemented and / or performed in accordance with any applicable owner / user manual and / or safety guidelines.

[0077] The disclosure has been described in an illustrative manner, and it is understood that the terminology used is intended to be in the nature of description rather than limitation. Many modifications and variations of the present disclosure are possible in light of the above teachings, and the disclosure may be practiced otherwise than as specifically described.

[0078] In the drawings, the same reference numerals indicate the same elements. Furthermore, some or all of these elements could be changed. With respect to the media, processes, systems, methods, etc. described herein, it is to be understood that although the steps of such processes, etc., have been described as occurring according to a certain ordered sequence, in carrying out such processes, the described steps could be performed in an order other than the order described herein, unless otherwise indicated or apparent from the context. Likewise, it is further understood that certain steps could be performed concurrently, other steps could be added, or certain steps described herein could be omitted.In other words, the descriptions of processes herein are for the purpose of illustrating certain examples and should not be construed in any way to limit the claims.

[0079] The adjectives first and second are used throughout this document as identifiers and are not intended to imply any meaning, order, or quantity unless explicitly stated otherwise.

[0080] The term exemplary is used herein to indicate an example, e.g., a reference to an exemplary device should be read merely as a reference to an example of a device.

[0081] The use of “in response to,” “based on,” and “when determined” in this paper indicates a causal relationship rather than a purely temporal one.

[0082] Computer-executable instructions may be compiled or interpreted from computer programs created using a variety of programming languages and / or technologies, including, without limitation, and either alone or in combination, Java, C, C++, Visual Basic, Java Script, Perl, Python, HTML, etc. In general, a processor, e.g., a microprocessor, receives instructions, e.g., from memory, a computer-readable medium, etc., and executes those instructions, thereby performing one or more processes that include one or more of the processes described in this document. Such instructions and other data may be stored and transmitted using a variety of computer-readable media. A file in a networked device is generally a collection of data stored on a computer-readable medium, such as a storage medium, random access memory, etc.are stored. A computer-readable medium includes any medium involved in providing data (e.g., instructions) that can be read by a computer. Such a medium can take many forms, including, without limitation, non-transitory media and volatile media. Instructions can be transmitted through one or more transmission media, including fiber optics, wires, wireless communications, including the internal structural elements comprising a system bus coupled to a processor of a computer. Common forms of computer-readable media include, for example, RAM, a PROM, an EPROM, a FLASH EEPROM, any other memory chip or cartridge, or any other medium from which a computer can read.

[0083] According to the present invention, a system is provided comprising: a computer having a processor and a memory, the memory storing instructions executable by the processor to: determine a presence of a passenger; determine a gaze direction of a vehicle operator based on an image of the vehicle operator; determine a position of a hand of the vehicle operator based on the image of the vehicle operator; and actuate a vehicle component based on the determined presence of the passenger, the determined gaze direction of the vehicle operator, and the determined position of the vehicle operator's hand.

[0084] According to one embodiment, the vehicle component is a steering component of a vehicle, and wherein the instructions further include instructions for actuating the steering component when the position of the hand is not in contact with a steering element of the vehicle and the instantaneous direction is away from a forward direction with respect to the vehicle.

[0085] According to one embodiment, the vehicle component is a human-machine interface (HMI) of a vehicle, and wherein the instructions further include instructions for activating the HMI when the passenger is present, the position of the hand is in contact with a steering element of the vehicle, and the eye direction is in a forward direction with respect to the vehicle.

[0086] According to one embodiment, the instructions further include instructions for deactivating the HMI when the position of the hand is not in contact with the steering element of the vehicle or the eye direction is not in the forward direction.

[0087] According to one embodiment, the instructions for determining a gaze direction include instructions for using the image of the vehicle operator to determine a gaze angle and map the gaze angle to a direction relative to a vehicle.

[0088] According to one embodiment, the instructions for determining a position of a hand of the vehicle operator include instructions for using the image of the vehicle operator to create a node model of the vehicle operator.

[0089] According to one embodiment, the instructions further include instructions for determining a gaze direction of the passenger based on an image of the passenger and determining a position of a hand of the passenger based on the image of the passenger.

[0090] According to one embodiment, the vehicle component is an MMS of a vehicle, and wherein the instructions further include instructions for activating the MMS when the passenger is present, the position of the vehicle operator's hand is in contact with a steering element of the vehicle, the vehicle operator's instantaneous direction is in a forward direction with respect to the vehicle, the passenger's instantaneous direction is toward the MMS, and the position of the passenger's hand is in contact with the MMS.

[0091] According to one embodiment, the instructions further include instructions for deactivating the MMS when the passenger is not present.

[0092] According to one embodiment, the MMS is a vehicle navigation system.

[0093] According to the present invention, a method includes: determining a presence of a passenger; determining a gaze direction of a vehicle operator based on an image of the vehicle operator; determining a position of a hand of the vehicle operator based on the image of the vehicle operator; and actuating a vehicle component based on the determined presence of the passenger, the determined gaze direction of the vehicle operator, and the determined position of the hand of the vehicle operator.

[0094] In one aspect of the invention, the vehicle component is a steering component of a vehicle and further comprising actuating the steering component when the position of the hand is not in contact with a steering element of the vehicle and the instantaneous direction is away from a forward direction with respect to the vehicle.

[0095] In one aspect of the invention, the vehicle component is a human-machine interface (HMI) of a vehicle and further comprising activating the HMI when the passenger is present, the position of the hand is in contact with a steering element of the vehicle, and the instantaneous direction is in a forward direction with respect to the vehicle.

[0096] In one aspect of the invention, the method includes deactivating the HMI when the position of the hand is not in contact with the steering element of the vehicle or the instantaneous direction is not in the forward direction.

[0097] In one aspect of the invention, determining a gaze direction includes using the image of the vehicle operator to determine a view angle and mapping the view angle to a direction relative to a vehicle.

[0098] In one aspect of the invention, determining a position of a hand of the vehicle operator includes using the image of the vehicle operator to create a node model of the vehicle operator.

[0099] In one aspect of the invention, the method includes determining a gaze direction of the passenger based on an image of the passenger and determining a position of a hand of the passenger based on the image of the passenger.

[0100] In one aspect of the invention, the vehicle component is an MMS of a vehicle and further comprising activating the MMS when the passenger is present, the position of the hand is in contact with a steering element of the vehicle, the vehicle operator's instantaneous direction is in a forward direction with respect to the vehicle, the passenger's instantaneous direction is toward the MMS, and the position of the passenger's hand is in contact with the MMS.

[0101] In one aspect of the invention, the method includes deactivating the MMS when the passenger is not present.

[0102] In one aspect of the invention, the MMS is a vehicle navigation system.

Claims

[1] System comprising: a computer including a processor and a memory, the memory storing instructions executable by the processor to: Determining the presence of a passenger; Determining a driver's instantaneous direction based on an image of the driver; Determining a position of a hand of the vehicle driver based on the image of the vehicle driver; and Actuating a vehicle component based on the determined presence of the passenger, the determined eye direction of the vehicle operator, and the determined position of the vehicle operator's hand. [2] The system of claim 1, wherein the vehicle component is a steering component of a vehicle, and wherein the instructions further include instructions for actuating the steering component when the position of the hand is not in contact with a steering element of the vehicle and the instantaneous direction is away from a forward direction with respect to the vehicle. [3] The system of claim 1, wherein the vehicle component is a human-machine interface (HMI) of a vehicle, and wherein the instructions further include instructions for activating the HMI when the passenger is present, the position of the hand is in contact with a steering element of the vehicle, and the eye-gaze direction is in a forward direction with respect to the vehicle. [4] The system of claim 3, wherein the instructions further include instructions for deactivating the HMI when the position of the hand is not in contact with the steering element of the vehicle or the eye direction is not in the forward direction. [5] The system of claim 1, wherein the instructions for determining a gaze direction include instructions for using the image of the vehicle operator to determine a view angle and map the view angle to a direction relative to a vehicle. [6] The system of any one of claims 1 to 5, wherein the instructions for determining a position of a hand of the vehicle operator include instructions for using the image of the vehicle operator to create a node model of the vehicle operator. [7] The system of claim 1, wherein the instructions further include instructions for determining a gaze direction of the passenger based on an image of the passenger and determining a position of a hand of the passenger based on the image of the passenger. [8] The system of claim 7, wherein the vehicle component is an HMI of a vehicle, and wherein the instructions further include instructions for activating the HMI when the passenger is present, the position of the vehicle operator's hand is in contact with a steering element of the vehicle, the vehicle operator's instantaneous direction is in a forward direction with respect to the vehicle, the passenger's instantaneous direction is toward the HMI, and the position of the passenger's hand is in contact with the HMI. [9] The system of claim 8, wherein the instructions further include instructions for deactivating the MMS when the passenger is not present. [10] The system of claim 8, wherein the MMS is a vehicle navigation system. [11] Procedure comprising: Determining the presence of a passenger; Determining a driver's instantaneous direction based on an image of the driver; Determining a position of a hand of the vehicle driver based on the image of the vehicle driver; and Actuating a vehicle component based on the determined presence of the passenger, the determined eye direction of the vehicle operator, and the determined position of the vehicle operator's hand. [12] The method of claim 11, wherein the vehicle component is a steering component of a vehicle and further comprising actuating the steering component when the position of the hand is not in contact with a steering element of the vehicle and the instantaneous direction is away from a forward direction with respect to the vehicle. [13] The method of claim 11, wherein the vehicle component is a human-machine interface (HMI) of a vehicle and further comprising activating the HMI when the passenger is present, the position of the hand is in contact with a steering element of the vehicle, and the instantaneous direction is in a forward direction with respect to the vehicle. [14] The method of claim 13, further comprising deactivating the HMI when the position of the hand is not in contact with the steering element of the vehicle or the instantaneous direction is not in the forward direction. [15] The method of any one of claims 11 to 14, wherein determining a position of a hand of the vehicle operator includes using the image of the vehicle operator to create a node model of the vehicle operator.

Images