Virtual interaction assistant for safe driving

Abstract

A virtual interaction assistant system for safe driving of an automobile. The system has one or more sensors, a display device, a storage medium comprising computer program instructions, and one or more processors coupled to communicate with the one or more sensors, the display device, and the storage medium. The one or more processors execute the instructions to cause the display device to display a see-through user interface on a driver's side of a windshield of the automobile. The user interface contains one or more selections. The one or more processors execute the instructions to receive sensor data from the one or more sensors of one or more hands of a human driving the automobile and determine a selection out of the one or more selections in the user interface based on the sensor data.

Description

VIRTUAL INTERACTION ASSISTANT FOR SAFE DRIVINGInventors: Jiechen Wang Xiyun Song Zongfang Lin Hong Heather Yu Yubin Zhou Zhiqiang Lao Liang PengFIELD

[0001] The disclosure generally relates to automobiles and, more specifically, to a system that supports command input techniques that improves driver safety.BACKGROUND

[0002] Driver distraction is increasingly becoming a leading cause of vehicular accidents. Automobile manufacturers are replacing traditional physical controls with touch screens, a modem and convenient technology integrated into vehicles' infotainment systems. These touch screens offer a sleek, interactive interface that allow drivers and passengers to control various functions and settings. They often serve as the central hub for navigation, entertainment, phone connectivity, climate control, and other vehicle-related features.

[0003] Increased complexity of touch screen display user interfaces, however, divert the driver’s attention away from the road. Preventing driver distraction is critical in assuring a safe driving environment not only for the driver and passengers, but also for other drivers in the vicinity that may be affected by the distracted driver.SUMMARY

[0004] One general aspect includes a virtual interaction assistant (VIA) system that improves safe driving of an automobile. An embodiment of the system comprises one or more sensors, a display device, a storage medium comprising computer programinstructions, and one or more processors coupled to communicate with the one or more sensors, the display device, and the storage medium. The one or more processors execute the instructions to cause the display device to display a see- through user interface on a driver’s side of a windshield of the automobile. The user interface contains one or more selections. The one or more processors execute the instructions to receive sensor data from the one or more sensors of one or more hands of a human driving the automobile. The one or more processors execute the instructions to determine a selection out of the one or more selections in the user interface based on the sensor data.

[0005] Implementations may include the VIA system where the display device is configured to display the see-through user interface on the windshield such that the see-through interface is within a line-of-sight of the human when looking forwards when driving the automobile. Implementations may include the VIA system where the one or more sensors are positioned to capture the sensor data of the one or more hands of the human driving the automobile while the human’s hands remain in contact with a steering wheel of the automobile. Implementations may include the VIA system where the one or more sensors comprise a camera having a field of view configured to capture an image of the human’s hands on a steering wheel of the automobile. Implementations may include the VIA system where the camera is located to capture an image of the human’s hands on a front surface of the steering wheel of the automobile, wherein the front surface faces the driver. Implementations may include the VIA system where the camera is located to capture an image of the human’s hands on a back surface of the steering wheel of the automobile, wherein the back surface faces away from the driver. Implementations may include the VIA system where the one or more sensors comprise a capacitive sensor integrated on a steering wheel of the automobile. The capacitive sensor is configured to sense interaction of the human’s hands with the steering wheel. Implementations may include the VIA system where the one or more sensors comprise one or more buttons integrated on a steering wheel of the automobile, the one or more buttons configured to detect a tap of a digit of the one or more hands of the human. Implementations may include the VIA system one or more processors that execute instructions to recognize gestures, motions, andtouches made by one or more hands of a human and determine interaction of the human’s hand with a see-through user interface based on the gestures. Implementations may include the VIA system wherein determining the interaction comprises determining whether the driver’s hand maps to an item being displayed in the see-through user interface. Implementations may include the VIA system wherein determining the interaction comprises determining whether the driver’s hand selects an item being displayed in the see-through user interface. Implementations may include the VIA system wherein the one or more processors execute the instructions to respond to the interaction by displaying a navigation page in the see-through user interface. Implementations may include the VIA system wherein the one or more processors execute the instructions to respond to the interaction by controlling a component of the automobile. Implementations may include the VIA system wherein the one or more sensors sense the driver’s hand gestures, hand motions, or hand touches of a steering wheel of the automobile and the one or more processors execute the instructions to analyze the sensed hand gestures, hand motions, or hand touches to: determine user interaction with the see-through user interface and control actions of the automobile based on the user interaction. Implementations may include the VIA system wherein the one or more processors execute the instructions to display results of the control actions in the see-through user interface. Implementations may include the VIA system the one or more sensors sense the driver’s hand gestures, hand motions, or hand touches of a steering wheel of the automobile and the one or more processors execute the instructions to analyze the sensed hand gestures, hand motions, or hand touches to: determine user interaction with the see-through user interface and map the sensed hand gestures, motions, or touches to selection of an item in the see-through user interface. Implementations may include the VIA system where the one or more processors execute the instructions to track landmarks of the one or more hands of the human driving the automobile and determine interaction of the human with the see-through user interface based on tracking the landmarks. Implementations may include the VIA system where one or more processors execute the instructions to map real-world physical coordinates of the one or more hands of the human to coordinates on the see-through user interface. Implementations may include the VIA system where the one or more processors execute the instructions toinstruct the display to present a cursor icon on the see-through user interface at the coordinates on the see-through user interface. Implementations may include the VIA system where the one or more processors execute the instructions to change a color of an item in the see-through user interface responsive the cursor icon being over the item. Implementations may include the VIA system where the one or more processors execute the instructions to change a color of an item in the see-through user interface responsive to selection of the item by the human. Implementations may include the VIA system where the one or more processors execute the instructions to control a component of the automobile responsive to the selection in the user interface.

[0006] Implementations may include the VIA system where the one or more processors execute the instructions to provide feedback to the human responsive to the human’s interaction with the user interface. Implementations may include the VIA system where the one or more processors execute the instructions to provide haptic feedback in a belt on the steering wheel responsive to the human’s interaction with the user interface. Implementations may include the VIA system where the one or more processors execute the instructions to provide a different strength of vibration in the belt for different interactions of the human with the see-through user interface. Implementations may include the VIA system where the one or more processors execute the instructions to adapt the see-through user interface based on the human’s past interactions with the see-through user interface. Implementations may include the VIA system where the one or more processors execute the instructions to instruct the display device to display the see-through user interface on the driver’s side of the windshield responsive to tapping of one or more digits of the human on a steering wheel of the automobile. Implementations may include the VIA system where the one or more processors execute the instructions to shut off the see-through user interface responsive to tapping of one or more digits of the human on a steering wheel of the automobile. Implementations may include the VIA system where the one more processors execute the instructions to adjust transparency of the see-through user interface.

[0007] One general aspect includes a method for providing a virtual interaction assistant for safe driving. The method comprises instructing a display device todisplay a see-through user interface on a driver’s side of a windshield of an automobile, the user interface contains one or more selections. The method comprises receiving, from one or more sensors, sensor data of one or more hands of a human driving the automobile. The method comprises determining a selection out of the one or more selections in the user interface based on the sensor data.

[0008] One general aspect includes a non-transitory computer-readable medium storing computer instructions that, when executed, are operable to provide a virtual interaction assistant for safe driving, and cause the one or more processors to: instruct a display device to display a see-through user interface on a driver’s side of a windshield of an automobile, the user interface contains one or more selections; receive, from one or more sensors, sensor data of one or more hands of a human driving the automobile; and determine a selection out of the one or more selections in the user interface based on the sensor data.

[0009] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter. The claimed subject matter is not limited to implementations that solve any or all disadvantages noted in the Background.BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Aspects of the present disclosure are illustrated by way of example and are not limited by the accompanying Figures for which like references indicate the same or similar elements.

[0011] FIG. 1 illustrates an example environment in which embodiments of a virtual interaction assistant (VIA) for safe driving may be practiced.

[0012] FIG. 2 is a block diagram of one embodiment of a VIA system, along with other elements in a vehicle.

[0013] FIG. 3A depicts an embodiment of sensors used to capture data of the driver’s hands.

[0014] FIG. 3B depicts an embodiment of a camera having a field of view that captures images of a back side (from the driver’s perspective) of the steering wheel.

[0015] FIG. 4 is a flowchart of one embodiment of a process of providing a VIA for safe driving of a vehicle.

[0016] FIG. 5 illustrates an example of a joint mapping using a type of hand land marking technology.

[0017] FIGs. 6A, 6B, and 6C show an example see-through user interface during operation of an embodiment of the VIA system.

[0018] FIG. 7 is a flowchart of an embodiment of a process of providing a VIA for safe driving.

[0019] FIG. 8 show an example layout in a see-through user interface (III) of an embodiment of the VIA system.

[0020] FIG. 9 shows another example layout for a top or main level page in a see- through III of an embodiment of the VIA system to illustrate personalization between drivers.

[0021] FIG. 10 show an example layout in a see-through III of an embodiment of the VIA system to illustrate driver personalization over time.

[0022] FIG. 11 is a flowchart of one embodiment of a process of providing a see- through III in an automobile windshield.

[0023] FIG. 12 is a flowchart of one embodiment of a process of validating a selected item in a see-through III.WRITTEN DESCRIPTION

[0024] The present disclosure and embodiments present systems and methods for a virtual interaction assistant (VIA) for safe driving. FIG. 1 illustrates an example environment in which embodiments of a virtual interaction assistant (VIA) for safe driving may be practiced. In general, FIG. 1 depicts a view from an interior of an automobile. The automobile has a touch screen 102 that offers an interactive interface to allows the driver and / or passenger to control various functions and settings. Driver interaction with the touch screen 102 poses safety risks that will be discussed in more detail below. An embodiment of VIA system provides a see-through user interface 104 on a driver’s side of the windshield of the automobile (also referred to as vehicle). The see-through user interface 104 is an image that is displayed on the windshield in a manner that allows the driver to see through the image. Moreover, the image serves as a user interface to allow the driver to interact with various components of the automobile. For example, the see-through user interface 104 allows the driver to control the same or similar functions as the touch screen 102 allows. In one aspect the windshield has built-in display capability wherein the see-through user interface 104 is built into the windshield. In another aspect a projector is used to project the see-through user interface 104 onto the windshield. The transparency of the see- through user interface 104 may be adjustable. Therefore, the see-through user interface 104 does not impair the driver’s view of the road and other objects outside of the vehicle. Also depicted are the hands 106a, 106b of the driver on the steering wheel 108. The driver may interact with the see-through user interface 104 without taking their hands off from the steering wheel 108. A driver’s interaction with the see- through user interface 104 refers to the driver’s hand actions (e.g., gestures, positions, motions, touch of steering wheel, etc.) with respect to the user interface 104 that enable the driver to operate and / or control the automobile and its functions. The VIA analyzes the hand actions and maps the driver’s hand actions to a region of the see- through user interface 104 in order to allow the driver to interact with the see-through user interface 104 without the driver taking either hand 106a, 106b off the steering wheel 108. Example interactions include the driver’s hand mapping to an item beingdisplayed in the see-through user interface 104 and the driver’s hand being used to select an item being displayed in the see-through user interface 104. For example, the driver can move the index finger of the right hand 106b to point to an item 110 on the see-through user interface 104 without taking either hand 106a, 106b off the steering wheel 108. Pointing to the item 110 may be used to, for example, control some function of the automobile such as climate control, windshield wipers, etc. Also, the driver may interact with the see-through user interface 104 without taking their eyes off the road. The VIA system may integrate into the automobile without occupying the position of components such as airbags, etc. The VIA system enhances safety without compromising existing vehicle structure.

[0025] The VIA system has one or more sensors to capture sensor data pertaining to one or more of the driver’s hands 106a, 106b. The sensor data may be for any part of the driver’s hands 106. Herein, the terms “finger” and “digit” are both used to refer to any of the five appendages of a human hand. Thus, the terms finger and digit, as used throughout this disclosure, include the thumb. FIG. 1 shows a camera 112 that has a field of view 114 to capture both of the user’s hands 106a, 106b. There may also be one or more sensors on the steering wheel 108 such as capacitive sensors in a belt 118 and / or buttons on the steering wheel 108. The capacitive sensors in the belt 118 and / or buttons may be used to allow the driver to turn the VIA system on and off, as well as make selections in the see-through user interface 104 without taking their hands off the steering wheel 108.

[0026] In one embodiment, the VIA system 200 maps between real-world coordinates of the driver’s hand to two-dimensional coordinates of the see-through user interface 104 and projects a cursor icon 116 on the see-through user interface 104. The cursor icon 116 in FIG. 1 has the shape of a hand, but another shape may be displayed. The VIA system 200 may analyze real-world coordinates of landmarks on a hand (e.g., right hand 106b) and / or gestures of the hand to map to the two- dimensional coordinates of the see-through user interface 104. The VIA system may recognize gestures made the driver’s hand to allow interaction with the see-through user interface 104. For example, the driver can select in item 110 on the see-through user interface 104 (an example of interaction with the see-through user interface 104)while keeping both hands 106a, 106b on the steering wheel and while maintaining their line of sight on the road.

[0027] In contrast, the touch screen 102 may pose safety risks. The touch screen 102 may serve as a central hub for navigation, entertainment, phone connectivity, climate control, and other vehicle-related features. The touch screen 102 serves in addition to, or as an alternative, to voice control. Voice control may be unsuitable in some cases such as a noisy environment.

[0028] Often, there will be no passenger to operate the touch screen 102 or the passenger may not be able to operate the touch screen. However, use of the touch screen 102 may pose significant safety concerns if used by the driver of the vehicle. A typical touch screen 102 may have many pages / groups of buttons, scrolling or sliding controls, and selectable items with different layouts on the touch screen 102. For example, there could be hundreds of touch selectable buttons on the different pages. Furthermore, the layout of the different pages will typically be different. Due to this complexity, it is very difficult to correctly interact with the touch screen 102 without looking at the contents on the touch screen 102. Therefore, the driver has to take his / her eyes away from the road ahead to interact with the touch screen 102. Furthermore, the driver also has to take their hand off from the steering wheel 108 to interact with the touch screen 102 for some duration of time. Because the vehicle is moving and the touch screen 102 lacks steady physical support for the hand such as the knobs and buttons in traditional cars, it is not easy for the finger to press at the right location on the touch screen 102. This is even more challenging for left-handed drivers sitting on the left side of the vehicle or right-handed drivers sitting on the right side of the vehicle. When mis-pressed, it may take extended time for the driver to continue to interact with the touch screen 102 instead of focusing on driving. Unfortunately, many automobile accidents happen within just one or two seconds of distraction. As one of many possible examples, if the windshield begins to fog up and there is not a passenger in the front seat the driver will need to take action to defog the windshield. The touch screen 102 may require the driver to press multiple items to bring up different pages to finally bring up a climate control display screen that enables window defogging. Thus, a need exists for a system that may be programmedto recognize user motions to activate specific commands to reduce the amount of user interactions with a touchscreen while driving the vehicle.

[0029] FIG. 2 is a block diagram of one embodiment of a VIA system 200, along with other elements in a vehicle. The VIA system 200 may be implemented within a vehicle such as the vehicle depicted in FIG. 1 . The VIA system 200 has a number of sensors 202. The sensors 202 may include, but are not limited to, one or more touch sensors 204, one or more visual sensors 206, and one or more microphones 208. The sensors 202 are configured to collect and provide data pertaining to the driver’s interaction with the VIA system 200. In an embodiment, the driver uses hand position, gestures, and / or actions, which the VIA system 200 analyzes to determine driver interaction with the see-through user interface 104. The sensors 202 are communicatively connected to processor 210 in order to provide the sensor data to the processor 210. The one or more touch sensors 204 may be integrated into the steering wheel 108. Examples of touch sensors 204 include, but are not limited to, capacitive bands / belts and buttons on the steering wheel 108. Examples of visual sensors 206 include, but are not limited to, RGB, infrared and thermal sensors. Optionally, a visual sensor 206 may have in-sensor computing capabilities to detect motions and perform some lightweight intelligent computations. Such lightweight intelligent computations may be helpful to reduce energy cost and data transmission to the processor 210.

[0030] The processor(s) 210 is communicatively connected to memory 212, to mass storage 214, and to transparently adjustable see-through display system 205. The processor(s) 210 may include a central processing unit (CPU). The processor(s) 210 may communicate with other components over a bus, which may be one or more of any type of several bus architectures. The processor(s) 210 may comprise any type of electronic data processor.

[0031] The memory 212 has stored therein computer program instructions which, when executed on the one or more processor(s) 210, perform various actions and functions of the VIA system 200. For ease of discussion the instructions are depicted as sub-blocks (or modules) within the memory 212. The modules include an ArtificialIntelligence (Al) engine for hand tracking and action recognition 216 (more briefly “Al engine”). The Al engine 216 identifies and tracks the landmarks of fingers in real time. The Al engine 216 may recognize the driver’s action based on the moves and gestures of the hand and fingers. This allows the driver to mimic actions the driver would use on a physical touch screen 102, such as navigation, swipe, scroll, slide, click, confirm, etc.

[0032] A group of the modules may be referred to the VIA control system 200. The display controller 218 controls the content displayed on the see-through display 104. The profile manager 220 manages driver profiles and preferences. The driver profiles and preferences 226 may be stored in the mass storage 214. The profile manager 220 may allow the driver to personalize the layout on the see-through display 104. The personalization may be based on explicit selection by the driver. The profile manager 220 may learn the driver’s habits, interaction with VIA system, etc. The personalization may be used to fine tune the Al model used by Al engine 216.

[0033] The feedback system 222 provides feedback to the driver regarding their interaction with the VIA system 200. The feedback could include, but is not limited to, visual on the see-through display 104, haptic via belt 118, and / or sound feedback via one or more speakers 228. Haptic feedback engages driver’s sense of control to enhance interactions with the see-through user interface (III) 104. Haptic feedback validates driver’s selection in the see-through III 104 by providing tangible responses. Haptic feedback can let the driver adjust their actions, confirms their input, reducing uncertainty and contributing to a more intuitive sense of control. Examples of visual feedback include, but are not limited to, highlighting an item on the see-through display 104 in response to the driver placing a finger over the item and / or selecting the item. The highlighting on the see-through display 104 may include changing the color of the item on the see-through display 104. Sound feedback may be used to, for example, confirm an action selected in the see-through display 104 such as “defogger is on”.

[0034] The data transfer manager 224 oversees transfer of data from the sensors 202. The VIA control system 217 is not limited to the example modules depicted in FIG. 2. Another example function performed by the VIA control system 217 isvalidation of actions associated with on item selected by the driver. The validation may integrate and sync with other operation systems of the automobile 230 to make sure all VIA actions performed by the driver are valid and allowed.

[0035] Memory 212 may comprise any type of system memory such as static random-access memory (SRAM), dynamic random-access memory (DRAM), synchronous DRAM (SDRAM), read-only memory (ROM), a combination thereof, or the like. In an embodiment, memory 212 may include ROM for use at boot-up, and DRAM for program and data storage for use while executing programs. In embodiments, the memory 212 is non-transitory. In one embodiment, the memory 212 includes computer-readable instructions that are executed by the processor(s) 210 to implement embodiments of the disclosed technology.

[0036] The mass storage device 214 may comprise any type of storage device configured to store data, programs, and other information and to make the data, programs, and other information accessible via the bus. The mass storage device 214 may comprise, for example, one or more of a solid-state drive, hard disk drive, a magnetic disk drive, an optical disk drive, or the like. The mass storage device 214 includes instructions which when executed by one or more processor 210 cause the processor(s) to perform the methods described herein. The mass storage 214 may include code in the form of application modules stored thereon, comprising instructions for causing the processor 210 to implement the components of VIA control system 217 and / or Al engine 216 which are illustrated as present in memory 212.

[0037] The processor 210 is communicatively connected to the physical touch screen 102 such that the processor 210 controls the physical touch screen 102. Therefore, at times the driver may interact with the see-through display 104; however, the physical touch screen 102 may still be used. The layouts provided in the see- through III 104 are not necessarily the same as the layouts provided in the physical touch screen 102. However, both user interfaces may be used to control various components in the automobile 230. Moreover, there may be software that interfaces with both the physical touch screen 102 and the see-through III 104 such that either of these user interfaces can be used to control various components in the automobile230. The processor 210 is communicatively connected to other components in the automobile, such that the processor 210 may receive input from the other components and / or control other components. These other components may be controlled by the driver through the VIA system 200.

[0038] FIG. 3A depicts an embodiment of sensors used to capture data of the driver’s hands. FIG. 3A shows a camera 112 positioned above the steering wheel 108. The camera 112 has a field of view 114 to enable capture of images of the driver’s hands 106a, 106b on the steering wheel 108. The camera 112 is located to capture an image of the human’s hands on a front surface of the steering wheel 108 of the automobile. The front surface of the steering wheel 108 faces the driver.

[0039] In an embodiment, the camera 112 captures a video stream. The camera 112 may include one or more RGB sensors and / or one or more infrared (IR) sensors. The different camera sensors could be used under different light settings. The RGB sensor(s) could be used under daytime light and the IR sensor(s) could be used when ambient light is at lower levels. The camera sensors may be used to sense the driver’s hand gestures, hand motions, hand positions, etc. The VIA system may analyze the driver’s hand gestures, hand motions, hand positions, etc. to determine user interaction with the see-through user interface 104 and to control actions of the automobile based on the user interaction.

[0040] A first belt 118a is integrated onto the steering wheel 108 in a location to interface with the driver’s left hand 106a. A second belt 118b is integrated onto the steering wheel 108 in a location to interface with the driver’s right hand 106b. Each belt 118 may have a touch sensitive sensor and / or a haptic feedback element. In an embodiment, one or both belts 118a, 118b contain a capacitive sensor. In an embodiment, one or both belts 118a, 118b contain a haptic feedback element. The capacitive sensor (or other touch based sensor) detect touches, taps, or the like of the steering wheel from the left hand 106a and / or the right hand 106b. The VIA system may analyze the driver’s touches, taps, or the like of the steering wheel to determine user interaction with the see-through user interface 104 and to control actions of the automobile based on the user interaction. The touch sensors in belts 118a, 118b mayallow the driver turn the VIA system 200 on and / or off while keeping their hands 106a, 106b on the steering wheel. The touch sensors may allow the driver to select an item in the see-through III 104 while keeping their hands 106a, 106b on the steering wheel. The haptic feedback element in the first belt 118a and / or the second belt 118b may vibrate to provide feedback to the driver.

[0041] A button 311 is integrated onto the steering wheel 108 in a location to detect touches, taps, or the like from the left hand 106a. Alternatively, or in addition to button 311 , a button could be integrated onto the steering wheel 108 in a location to detect touches, taps, or the like from the right hand 106b. There may be more than one button 311 for a particular hand to allow different functionality for buttons. The one or more buttons 311 may allow the driver turn the VIA system 200 on and / or off while keeping their hands 106a, 106b on the steering wheel. The one or more buttons 311 may allow the driver to make a selection in the see-through III 104 while keeping their hands 106a, 106b on the steering wheel.

[0042] FIG. 3B depicts an embodiment of a camera 302 having a field of view 314 that captures images of a back side (from the driver’s perspective) of the steering wheel 108. The camera 302 is positioned to capture images of the driver’s fingers 306a, 306b on the back surface of the steering wheel 108. The back surface of the steering wheel 108 faces away from the driver. Therefore, the clear view of the fingers 306a, 306b improves motion detection. The motion detection may be used to detect gestures such as, but not limited to, typing, sliding, etc.

[0043] FIG. 4 is a flowchart of one embodiment of a process 400 of providing a VIA for safe driving of a vehicle. The process 400 may be performed by VIA system 200. Step 402 includes instructing a transparency adjustable see-through display device 205 to display a see-through user interface 104 on a driver’s side of a windshield of an automobile. The see-through user interface 104 provides at least one selectable item, icon or the like. For brevity, the term item will be used. Typically, there may be multiple items in the see-through user interface 104.

[0044] Step 404 includes collecting sensor data of at least one hand of a human driver of the automobile. The sensor data may be collected from one or more cameras(e.g., camera 112, camera 302), one or more capacitor sensors (e.g., capacitive sensors in belts 118a, 118b), one or more buttons (e.g., button 311 ), but is not limited to these sensors. The cameras could be, for example, RGB or IR.

[0045] Step 406 includes determining a driver selection in the see-through user interface 104 based on the sensor data of the at least one hand. In response to the driver selection the VIA system 200 may take an action such as displaying a different page in the see-through III 104 or instructing a component in the automobile to perform an action (e.g., turn on defogger).

[0046] As noted above, the VIA system 200 may have an Al engine 216. The Al engine 216 may be used for finger tracking and gesture recognition. Example gestures that may be detected by the Al engine 216 include, but are not limited to, “click to select,” “click to confirm,” “sliding bar to adjust value,” “move to the next item on the left, right, upper or lower,” “return back to previous page,” “cancel or escape (ESC),” “done, turn off VIA system.” Some of these gestures may be combined with the capacitive sensor in belt 118, buttons 311 on the steering wheel 108, and / or voice control. The Al engine 216 may learn the driver’s finger characteristics and operation habits, such that the driver’s profile may be updated to improve finger tracking and gesture recognition. For example, each driver may make gestures in a slightly different manner. Also, drivers may have different size and shape of hands; therefore, the landmarks tracked by the Al engine may vary between drivers.

[0047] The Al engine 216 is able to identify landmarks of the driver’s hands. The Al engine 216 may implement a hand pose estimation algorithm. A hand pose estimation algorithm may be used to estimate the hand pose and provide real-world coordinates (e.g., X-, Y- Z-coordinates) of the landmarks. In an embodiment, the real- world coordinates of the landmarks are mapped to X-, Y-coordinates of the see- through III 104. FIG. 5 illustrates one possible landmarking of key points for hand pose estimation. The Al engine 216 may learn the landmarks specific to a driver and store a model specific to the driver in the user profile and preferences database 226. Each of the landmarks illustrated in FIG. 5 can move in two or three physical dimensions, and a time domain. In embodiments herein, an input may comprise asingle image frame from a camera, with the output being a normalized 3D coordinate (X, Y, and Z) of 21 joints for one or both hands. As cameras provide video data, each video (including multiple frames), a 1 D time series data in each axis for each joint which can be processed (e.g., filtered or smoothed) to improve consistency and robustness. Various hand pose estimation algorithms are known to those of ordinary skill in the art. One example hand pose estimation algorithm is MediaPipe Hand Landmarker provided by Google Inc. of Mountain View, California. However, the Al engine 216 is not limited to this hand pose estimation algorithm.

[0048] FIGs. 6A, 6B, and 6C show an example see-through user interface 104 during operation of an embodiment of the VIA system 200. Note that for clarity of illustration the transparent aspect of the see-through user interface 104 is not depicted in FIGs. 6A - 6C. However, it will be understood that when the see-through user interface 104 of FIGs. 6A - 6C is projected on a windshield of an automobile the driver will be able to see-through the user interface 104. The transparency of the see- through the user interface 104 may be adjusted based on lighting conditions such that the driver is always able to easily see through the user interface 104 to maintain clear vision of the road.

[0049] Referring now to FIG. 6A, the see-through user interface 104 has a number of selectable items 602a, 602b, 602c, 602d, and 602e. The VIA system 200 has analyzed the location, gestures, and / or movements of the driver’s right hand (not depicted in FIG. 6A) and projected a cursor icon 116 on the see-through user interface 104. In an embodiment, the cursor icon 116 has an appearance of a human hand, but other shapes may be used for the cursor icon 116. The VIA system 200 may factor in the real-world coordinates of landmarks of the driver’s hand to determine where to project the cursor icon 116. In an embodiment, the VIA system 200 tracks a particular part on the hand such as a fingertip. In an embodiment, the real-world coordinates of the landmarks are mapped to X-, Y-coordinates of the see-through III 104 in order to display the cursor icon 116. In an embodiment, the driver may move their fingertip to cause the cursor icon 116 to move. Thus, the driver is able to control the location of the cursor icon 116 without taking their hand off from the steering wheel. In FIG. 6A the fingertip 616 of the cursor icon 116 is not over any of the items 602. Referring nowto FIG. 6B, the driver has moved their hand and / or finger such that the cursor icon 116 is now displayed at a location in which the fingertip 616 of the cursor 116 is over item 602b. The VIA system 200 may highlight item 602b responsive to the fingertip 616 of the cursor icon 116 being over item 602b. Optionally, the VIA system 200 may highlight item 602b responsive to the fingertip 616 in the cursor icon 116 hovering over item 602b for a pre-determined period of time. The driver may establish the “hover time” in their driver profile.

[0050] The driver may make an additional action to select item 602b. FIG. 6C shows that the VIA system 200 has changed the appearance of item 602b in response to the driver selecting item 602b. In one embodiment, the VIA system 200 changes the color used to display an item 602 in order to indicate an over event (FIG. 6B) or a driver selection (FIG. 6C). The driver’s selection may be made in a variety of ways. For example, the driver’s selection may be made by the driver interacting with a touch sensor 204, such as capacitive sensor in belt 118 or a button 311 on the steering wheel 108. The driver’s selection may be made by a voice command, wherein a microphone 208 receives the driver’s selection. The driver’s selection may be made by a hand gesture, which may be interpreted by the Al engine 216. Although a cursor icon 116 is depicted in the example in FIGs. 6A - 6C, the VIA system 200 may display a different icon such as a cursor or another object that associates the driver’s real- world hand position and to a location on the see-through III 104.

[0051] FIG. 7 is a flowchart of an embodiment of a process 700 of providing a VIA for safe driving. Process 700 provides further details for an embodiment of process 400 in FIG. 4. Process 700 may be performed by the VIA system 200. Step 702 includes receiving a trigger to activate the VIA from the driver of the vehicle. This trigger could include, for example, a voice command, a hand gesture, a finger tap on the steering wheel, etc.

[0052] Step 704 includes activating VIA components in response to the trigger. Optionally, at least some of the sensors 202 may be inactive when the driver is not actively using the see-through III 104. For example, the visual sensors 206 may beinactive when the driver is not actively using the see-through III 104, wherein the visual sensors 206 are activated in step 704.

[0053] Step 706 includes displaying the see-through III 104 on the driver’s side of the windshield of the vehicle. Optionally, step 706 could be initiated after step 708 and / or 710.

[0054] Step 708 includes the visual sensor(s) 206 capturing data of the driver’s hands. The visual sensor(s) 206 may capture one or more video streams of the driver’s hands. The vehicle could be driven by different people who may place their hands in slightly different locations on the steering wheel 108, have different hand sizes and shapes, etc. The driver’s profile may be used to determine how best to capture the driver’s hands with the visual sensors.

[0055] Step 710 includes the Al engine 216 identifying and tracking the driver’s hands. Tracking the hands includes tracking any portion of the hand such as the fingers. The Al engine 216 may track various landmarks on the drivers hand(s). The driver’s profile may be used to determine how best to identify and track the driver’s hands. The Al engine 216 could track one finger in particular. The finger that is tracked may be part of the driver’s profile such that the driver has the choice of which finger to use to interact with the see-through III 104.

[0056] Step 712 includes displaying an icon representing the driver’s hand and / or finger overlaid on the see-through III 104 on the windshield. For example, a cursor icon 116 may show one of the driver’s fingers in a pointing configuration. Step 712 may include mapping real-world coordinates of landmarks of the driver’s hand to coordinates in the see-through III 104.

[0057] Step 714 includes a determination of whether the finger of the driver’s hand is over an item in the see-through II I 104. This determ ination of whether the finger of the driver’s hand is over an item in the see-through III 104 is based on the mapping of the real-world coordinates of the landmarks of the driver’s hand to the coordinates in the see-through III 104. If the finger is not over an item, the VIA system 200 continues to display the cursor icon 116 in the see-through III 104. If the VIA system200 determines that the driver has a finger over an item in the see-through III 104, then the item 602 is highlighted in step 716. Step 714 is an example of determining interaction of the driver with the see-through III 104.

[0058] Step 718 includes a determ ination of whether the driver has selected the item 602. Step 714 is an example of determining interaction of the driver with the see- through III 104. If the driver has not selected the item 602 then the VIA system 200 may continue to determine whether the driver’s finger is still over the item. If the finger is no longer over the item 602, then the VIA system 200 may stop highlighting the item 602 while continuing to display the cursor icon (step 712). However, if the driver’s finger is still over the item 602, then the VIA system 200 may continue highlighting the item 602 (step 716). If the VIA system 200 determines that the driver has selected the item 602, then in step 720 the VIA system 200 performs an action in accordance with the selected item. Step 720 could include changing the content in the see-through III 104 to, for example, show another page in the III. Step 720 could include the VIA system 200 instructing some other component in the vehicle to perform some action, such as to turn on the windshield defogger.

[0059] In an embodiment, the profiler manager 220 tailors the layout of the see- through III 104 to the driver based on the driver’s preference, habit, and / or frequency of use. FIG. 8 show an example layout in a see-through III 104 of an embodiment of the VIA system 200. The layout may be for a top or main level page. The driver may select one of the items 802a, 802b, 802c, 802d, 802e, or 802f to cause the VIA system 200 to display a lower level page that is specific to the selected item. In the example in FIG. 8, the layout includes selectable items for “Bluetooth,” “Charge,” “Contacts,” “Climate,” “Navigation,” and “Trips, respectively.” In an embodiment, the items themselves and their location in the layout are stored in the driver profile and preferences 226 for the driver. FIG. 9 shows another example layout for a top or main level page. As an example, the layout in FIG. 9 may be tailored to a different driver. In the example in FIG. 9, the layout includes selectable items 902a, 902b, 902c, 902d, 902e, 902f for “Climate,” “Navigation,” “Wiper,” “Phone,” “Media Source,” and “Charge, respectively.” Note that the layout in FIG. 9 contains a different set of items 902 than the items 802 in the layout in FIG. 8. Also note that Climate item 902a, Navigationitem 902b, and Charge Item 902f in the layout of FIG. 9 are in different locations relative to the Climate item 802d, Navigation item 802e, and Charge Item 802b in the layout of FIG. 8.

[0060] In an embodiment, the VIA system 200 modifies the layout that is presented to the see-through III 104 to the driver to personalize the layout to the driver. FIG. 10 show items 1002a, 1002b, 1002c, 1002d, 1002e, and 1002f in an example layout in a see-through III 104 of an embodiment of the VIA system 200. The layout may be for a top or main level page. The example in FIG. 10 shows a modification made to the layout in FIG 9 in order to personalize the layout to the driver. In the example in FIG. 10, the VIA system has switched the locations of “Climate” and “Navigation.” Also, “Charge” has been replaced by “Trips.” Thus, the VIA system 200 may move or replace items in the layout to personalize the layout to the driver. The personalization of the driver’s profile may include, but is not limited to, display layout, display location and size, font size, display transparency, and / or mapping speed for finger movements. In an embedment, the Al engine 216 maintains and updates a finger gesture model for the driver.

[0061] Note that for clarity of illustration the transparent aspect of the see-through user interface 104 is not depicted in FIGs. 8, 9 and 10. However, it will be understood that when the see-through user interface 104 of FIGs. 8 - 10 is projected onto a windshield of an automobile the driver will be able to see-through the user interface 104. The transparency of the see-through the user interface 104 may be adjusted based on lighting conditions such that the driver is always able to easily see through the user interface 104 to maintain clear vision of the road.

[0062] FIG. 11 is a flowchart of one embodiment of a process 1100 of the VIA system 200 providing a see-through III 104 in an automobile windshield. The process 1100 may be performed by the VIA system 200. The process 1100 provides further details of an embodiment of steps 712 - 720 in process 700. Thus, process 1100 begins after the VIA system 200 has been activated and a first page is displayed in the see-through III 104. The VIA system 200 may also begin to collect sensor data and track the driver’s hands upon system activation. An example in which the driverfirst selects the climate control and then selects the window defogger will be discussed. However, process 1100 has more general application.

[0063] Step 1102 includes displaying a cursor icon overlaid on a first page of a layout in the see-through III 104. For the sake of example the first page may be the main page depicted in the layout of the see-through III 104 in FIG. 8. An example will be discussed in which the driver selects the item “Climate.” In step 1102, a cursor icon (similar to cursor icon 116 in FIG. 6A) may be displayed in the layout.

[0064] Step 1104 includes receiving sensor data of the driver moving an index finger in the air with the hands still on the steering wheel. With reference to FIG. 3A the index finger of the driver’s right hand 106b is in the air while the driver’s right hand 106b is still on the steering wheel 108. Moreover, the driver’s right hand 106b maintains control of the steering wheel 108.

[0065] Step 1106 includes a determination of whether the fingertip moves over an item in the see-through III 104. The phrase “the fingertip moves over an item in the see-through III 104” refers to the concept of mapping between the real-world coordinates of the landmarks of the driver’s hand to the coordinates of the see-through III 104. This determination may be alternatively stated as a determination of whether the cursor icon 116 (or fingertip of the cursor icon 116) is over an item 602 in the see- through III 104. Step 1108 includes highlighting the item in the see-through III 104 in response to the fingertip moving over the item. As an example, the driver moves their fingertip such that the VIA system 200 displays the fingertip of a cursor icon over the climate item.

[0066] Step 1110 includes receiving a “click” gesture to select the item. The click gesture may be made by the driver in a number of ways. Also different types of sensors may be used to detect the click gesture.

[0067] Step 1112 includes changing the see-through III 104 to a second page. As an example, selection of the climate item causes the VIA system 200 to display the page depicted in FIG. 6A. The VIA system 200 may also display a cursor icon, similar to the cursor icon 116 in FIG. 6A.

[0068] Step 1114 includes determining whether the fingertip of the cursor icon 116 moves over an item in the second page of the see-through III 104.

[0069] Step 1116 includes highlighting an item in the second page of the see- through III 104 responsive to the fingertip of the cursor icon 116 moving over the item. With reference to FIG. 6B, the defog item 602b is highlighted in response to the fingertip of the cursor icon 116 moving over item 602b. The VIA system 200 therefore provides a form of visual feedback to the driver to confirm the location of the driver’s finger relative to the items on the see-through III 104.

[0070] Step 1118 includes receiving a “click” gesture to select the item in the second page. The “click” gesture may be made in a variety of ways by the driver. One example is for the driver to make a motion with their finger to simulate the action a person would do to touch an item in a touch screen. However, the “click” gesture is not required to be made with the finger that is being represented in the cursor icon 116. For example, the driver could press the capacitive sensor in belt 118 or button 311 to make the click gesture. The “click” gesture could be a verbal command from the driver.

[0071] Step 1120 includes validating the selected item. FIG. 12 provides further details of validating the selected item.

[0072] FIG. 12 is a flowchart of one embodiment of a process 1200 of validating a selected item in a see-through III 104. The process 1200 provides further details of an embodiment of step 1120 in process 1100. Step 1202 includes determining whether a valid selection has been made.

[0073] Steps 1204 and 1206 may be performed if the selection is valid. The VIA system 200 highlights the selected item in step 1204 responsive to the selection being valid. FIG. 6C depicts an example, in which the defog item 602b is highlighted in a different manner than in FIG., 6B to provide visual feedback of the selection to the driver. In step 1206 an action is performed for the selected item. As one example, the automobile’s windshield defogger is turned on. The VIA system 200 may communicate with other components in the automobile to effectuate step 1206.

[0074] However, if the selection is invalid then steps 1208 - 1212 may be performed. Steps 1208 - 1212 are one possible sequence for an invalid selection; however, other sequences are possible. Step 1208 includes providing a warning message to the driver. As an example, a message may be displayed in the see- through III 104 that the defogger may not be used at this time. Step 1210 includes receiving a back selection from the driver. For example, the driver makes a “back gesture” with one of their hands. Step 1212 includes returning to the first page in the see-through III 104. As an example, the VIA system 200 may return to presenting the main page in FIG. 8 on the see-through III 104. Feedback may also be provided to the driver in step 1212. The driver may then again make selections in the first page of the see-through III 104.

[0075] At some point the VIA system may discontinue displaying the see-through III 104 on the windshield. Step 1214 includes the VIA system 200 determining that one or more components in the VIA system 200 are to be inactivated. In one aspect the trigger may be an explicit command from the driver. The explicit command from the driver could be provided by voice, button 311 , capacitive sensor in belt 118, hand gesture, etc. In one aspect the trigger may be inactivity for a timeout period. The trigger could be the successful action in step 1206, such as turning on the defogger. Step 1216 is to step one of more components in the VIA system 200. Step 1216 may include, but is not limited to, discontinuing the display of the see-through III 104 on the windshield, shutting down sensors (e.g., visual sensors 206, touch sensors 204 and / or microphones 208), shutting down the Al engine 216, discontinuing execution of some or all of the code for the VIA control system 217 executed on processor 210.

[0076] For the purposes of this document, it should be noted that the dimensions of the various features depicted in the figures may not necessarily be drawn to scale.

[0077] For purposes of this document, reference in the specification to “an embodiment,” “one embodiment,” “some embodiments,” or “another embodiment” may be used to describe different embodiments or the same embodiment.

[0078] For the purposes of this document, a connection may be a direct connection or an indirect connection (e.g., via one or more other parts). In some cases, when anelement is referred to as being connected or coupled to another element, the element may be directly connected to the other element or indirectly connected to the other element via intervening elements. When an element is referred to as being directly connected to another element, then there are no intervening elements between the element and the other element. Two devices are “in communication” if they are directly or indirectly connected so that they can communicate electronic signals between them.

[0079] Although the present disclosure has been described with reference to specific features and embodiments thereof, it is evident that various modifications and combinations can be made thereto without departing from the scope of the disclosure. The specification and drawings are, accordingly, to be regarded simply as an illustration of the disclosure as defined by the appended claims, and are contemplated to cover any and all modifications, variations, combinations, or equivalents that fall within the scope of the present disclosure.

[0080] The technology described herein can be implemented using hardware, software, or a combination of both hardware and software. The software used is stored on one or more of the processor readable storage devices described above to program one or more of the processors to perform the functions described herein. The processor readable storage devices can include computer readable media such as volatile and non-volatile media, removable and non-removable media. By way of example, and not limitation, computer readable media may comprise computer readable storage media and communication media. Computer readable storage media may be implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Examples of computer readable storage media include RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information, and which can be accessed by a computer. A computer readable medium or media does (do) not include propagated, modulated, or transitory signals.

[0081] Communication media typically embodies computer readable instructions, data structures, program modules or other data in a propagated, modulated, or transitory data signal such as a carrier wave or other transport mechanism and includes any information delivery media. The term “modulated data signal” means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as RF and other wireless media. Combinations of any of the above are also included within the scope of computer readable media.

[0082] In alternative embodiments, some or all of the software can be replaced by dedicated hardware logic components. For example, and without limitation, illustrative types of hardware logic components that can be used include Field-programmable Gate Arrays (FPGAs), Application-specific Integrated Circuits (ASICs), Applicationspecific Standard Products (ASSPs), System-on-a-chip systems (SOCs), Complex Programmable Logic Devices (CPLDs), special purpose computers, etc. In one embodiment, software (stored on a storage device) implementing one or more embodiments is used to program one or more processors. The one or more processors can be in communication with one or more computer readable media / storage devices, peripherals and / or communication interfaces.

[0083] It is understood that the present subject matter may be embodied in many different forms and should not be construed as being limited to the embodiments set forth herein. Rather, these embodiments are provided so that this subject matter will be thorough and complete and will fully convey the disclosure to those skilled in the art. Indeed, the subject matter is intended to cover alternatives, modifications, and equivalents of these embodiments, which are included within the scope and spirit of the subject matter as defined by the appended claims. Furthermore, in the following detailed description of the present subject matter, numerous specific details are set forth in order to provide a thorough understanding of the present subject matter. However, it will be clear to those of ordinary skill in the art that the present subject matter may be practiced without such specific details.

[0084] Aspects of the present disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses (systems) and computer program products according to embodiments of the disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general-purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable instruction execution apparatus, create a mechanism for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0085] The description of the present disclosure has been presented for purposes of illustration and description but is not intended to be exhaustive or limited to the disclosure in the form disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the disclosure. The aspects of the disclosure herein were chosen and described in order to best explain the principles of the disclosure and the practical application, and to enable others of ordinary skill in the art to understand the disclosure with various modifications as are suited to the particular use contemplated.

[0086] For purposes of this document, each process associated with the disclosed technology may be performed continuously and by one or more computing devices. Each step in a process may be performed by the same or different computing devices as those used in other steps, and each step need not necessarily be performed by a single computing device.

[0087] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

Claims

CLAIMSWhat is claimed is:1 . A virtual interaction assistant (VIA) system for improving safety while driving an automobile, the system comprising: one or more sensors; a display device; a storage medium comprising computer program instructions; and one or more processors coupled to communicate with the one or more sensors, the display device, and the storage medium, wherein the one or more processors execute the instructions to: cause the display device to display a see-through user interface on a driver’s side of a windshield of the automobile, the user interface contains one or more selections; receive sensor data from the one or more sensors of one or more hands of a human driving the automobile; and determine a selection out of the one or more selections in the user interface based on the sensor data.

2. The system of claim 1 , wherein the display device is configured to display the see-through user interface on the windshield such that the see-through user interface is within a line-of-sight of the human when looking forwards when driving the automobile.

3. The system of any of claims 1 to 2, wherein the one or more sensors are positioned to capture the sensor data of the one or more hands of the human driving the automobile while the human’s hands remain in contact with a steering wheel of the automobile.

4. The system of any of claims 1 to 3, wherein the one or more sensors comprise: a camera having a field of view configured to capture an image of the human’s hands on a steering wheel of the automobile.

5. The system of any of claims 1 to 4, wherein the camera is located to capture an image of the human’s hands on a front surface of the steering wheel of the automobile, wherein the front surface faces the driver.

6. The system of any of claims 1 to 4, wherein the camera is located to capture an image of the human’s hands on a back surface of the steering wheel of the automobile, wherein the back surface faces away from the driver.

7. The system of any of claims 1 to 6, wherein the one or more sensors comprise a capacitive sensor integrated on a steering wheel of the automobile, the capacitive sensor configured to sense interaction of the human’s hands with the steering wheel.

8. The system of any of claims 1 to 7, wherein the one or more sensors comprise: one or more buttons integrated on a steering wheel of the automobile, the one or more buttons configured to detect a tap of a digit of the one or more hands of the human.

9. The system of any of claims 1 to 8, wherein the one or more processors execute the instructions to: recognize gestures made by the one or more hands of the human; and determine interaction of the human’s hand with the see-through user interface based on the gestures.

10. The system of claim 9, wherein determining the interaction comprises determining whether the driver’s hand maps to an item being displayed in the see- through user interface.11 . The system of claim 9, wherein determining the interaction comprises determining whether the driver’s hand selects an item being displayed in the see- through user interface.

12. The system of claim 9, wherein the one or more processors execute the instructions to respond to the interaction by displaying a navigation page in the see-through user interface.

13. The system of claim 9, wherein the one or more processors execute the instructions to respond to the interaction by controlling a component of the automobile.

14. The system of claim 9, wherein the one or more processors execute the instructions to generate an output to the see-through user interface responsive to the interaction.

15. The system of any of claims 1 to 14, wherein: the one or more sensors sense the driver’s hand gestures, hand motions, or hand touches of a steering wheel of the automobile; and the one or more processors execute the instructions to analyze the sensed hand gestures, hand motions, or hand touches to: determine user interaction with the see-through user interface; and control actions of the automobile based on the user interaction.

16. The system of claim 15, wherein the one or more processors execute the instructions to: display results of the control actions in the see-through user interface.

17. The system of any of claims 1 to 14, wherein: the one or more sensors sense the driver’s hand gestures, hand motions, or hand touches of a steering wheel of the automobile; and the one or more processors execute the instructions to analyze the sensed hand gestures, hand motions, or hand touches to: determine user interaction with the see-through user interface; and map the sensed hand gestures, motions, or touches to selection of an item in the see-through user interface.

18. The system of any of claims 1 to 17, wherein the one or more processors execute the instructions to: track landmarks of the one or more hands of the human driving the automobile; and determine interaction of the human with the see-through user interface based on tracking the landmarks.

19. The system of any of claims 1 to 18, wherein the one or more processors execute the instructions to: map real-world physical coordinates of the one or more hands of the human to coordinates on the see-through user interface.

20. The system of claim 19, wherein the one or more processors execute the instructions to: instruct the display to present a cursor icon on the see-through user interface at the coordinates on the see-through user interface.21 . The system of claim 20, wherein the one or more processors execute the instructions to change a color of an item in the see-through user interface responsive the cursor icon being over the item.

22. The system of any of claims 1 to 21 , wherein the one or more processors execute the instructions to change a color of an item in the see-through user interface responsive to selection of the item by the human.

23. The system of any of claims 1 to 22, wherein the one or more processors execute the instructions to: provide feedback to the human responsive to the human’s interaction with the user interface.

24. The system of any of claims 1 to 23, wherein the one or more processors execute the instructions to provide haptic feedback in a belt on the steering wheel responsive to the human’s interaction with the user interface.

25. The system of claim 24, wherein the one or more processors execute the instructions to provide a different strength of vibration in the belt for different interactions of the human with the see-through user interface.

26. The system of any of claims 1 to 25, wherein the one or more processors execute the instructions to adapt the see-through user interface based on the human’s past interactions with the see-through user interface.

27. The system of any of claims 1 to 26, wherein the one or more processors execute the instructions to validate a selection in the see-through user interface made by the human.

28. The system of any of claims 1 to 27, wherein the one or more processors execute the instructions to instruct the display device to display the see-through user interface on the driver’s side of the windshield responsive to tapping of one or more digits of the human on a steering wheel of the automobile.

29. The system of any of claims 1 to 22, wherein the one or more processors execute the instructions to shut off the see-through user interface responsive to tapping of one or more digits of the human on a steering wheel of the automobile.

30. The system of any of claims 1 to 29, wherein the one or more processors execute the instructions to adjust transparency of the see-through user interface.31 . The system of any of claims 1 to 30, wherein the one or more processors execute the instructions to control a component of the automobile responsive to the selection in the user interface.

32. A method for providing a virtual interaction assistant for safe driving, the method comprising: instructing a display device to display a see-through user interface on a driver’s side of a windshield of an automobile, the user interface contains one or more selections; receiving, from one or more sensors, sensor data of one or more hands of a human driving the automobile; and determining a selection out of the one or more selections in the user interface based on the sensor data.

33. A non-transitory computer-readable medium storing computer instructions to provide a virtual interaction assistant for safe driving, that when executed by one or more processors, cause the one or more processors to: instruct a display device to display a see-through user interface on a driver’s side of a windshield of an automobile, the user interface contains one or more selections; receive, from one or more sensors, sensor data of one or more hands of a human driving the automobile; and determine a selection out of the one or more selections in the user interface based on the sensor data.

Images