Multi-modal virtual assistant

Abstract

Methods and systems are provided that include one or more first sensors of a vehicle, one or more second sensors, and a processor. The one or more first sensors have a first modality and are configured to receive a first input related to the request from an occupant of the vehicle. The processor is configured to at least facilitate providing instructions to the passenger for providing additional inputs related to the request for a predetermined amount of time. The one or more second sensors have a second modality different from the first modality and are configured to receive a second input related to the request from the passenger. The processor is further configured to at least facilitate: interpreting the second input; and performing a vehicle action corresponding to the request based on the interpretation of the second input.

Description

TECHNICAL FIELD

[0001] The technical field is generally related to platforms such as vehicles, and more specifically, to methods and systems for facilitating interaction with passengers of a vehicle via a virtual assistant. BACKGROUND

[0002] Many vehicles today utilize technology for interacting with passengers of the vehicle. However, in certain situations, such technology can not always be optimal.

[0003] Accordingly, it is desirable to provide improved methods and systems for facilitating interaction with passengers, such as for vehicles. Furthermore, other desirable features and characteristics of the present disclosure will become apparent from the subsequent detailed description and the appended claims, taken in conjunction with the accompanying drawings and the foregoing technical field and background. SUMMARY

[0004] In an example embodiment, a method is provided that includes receiving, via one or more first sensors of a vehicle, a first input from a passenger of the vehicle related to a request, the one or more first sensors having a first modality; providing, via a processor of the vehicle, an instruction to the passenger for providing an additional input related to the request within a predetermined amount of time; receiving, via one or more second sensors of the vehicle, a second input from the passenger related to the request within the predetermined amount of time in response to the instruction, the one or more second sensors having a second modality different from the first modality; interpreting, via the processor, the second input; and performing, via the processor, a vehicle action corresponding to the request based on the interpretation of the second input.

[0005] Further in an example embodiment, the predetermined amount of time is determined via the processor based on prior history via adaptive learning.

[0006] Further in an example embodiment, the first input includes a voice command from the passenger and is received via one or more microphones of the vehicle.

[0007] Further in an example embodiment, the instruction includes an audio instruction provided via a speaker of the vehicle coupled to the processor.

[0008] Further in an example embodiment, the instruction includes a visual instruction provided via a display screen of the vehicle coupled to the processor.

[0009] Further in an example embodiment, the instruction informs the passenger to engage a particular input device in a particular directional manner within the predetermined amount of time based at least in part on a proximity of the passenger to the particular input device; and the second input is received via the one or more input sensors with respect to engagement of the particular input device in the particular directional manner within the predetermined amount of time.

[0010] Further in example embodiments, the instruction informs the passenger to engage a particular input device that is typically used for the first vehicle function; and the second input is received via the one or more input sensors with respect to engagement of the input device for performing the request with respect to a second vehicle function that is different from and unrelated to the first vehicle function.

[0011] Further in example embodiments, the instruction informs the passenger to perform a particular gesture within a predetermined amount of time that is unrelated to any input device of the vehicle; and the second input is received via the one or more cameras with respect to the particular gesture within the predetermined amount of time.

[0012] Further in example embodiments, the instruction informs the passenger to swipe a steering wheel of the vehicle via a hand or finger of the passenger within a predetermined amount of time; and the second input is received via the one or more cameras with respect to the swipe of the steering wheel of the vehicle via the hand or finger of the passenger within the predetermined amount of time.

[0013] In another example embodiment, a system is provided that includes one or more first sensors of a vehicle, one or more second sensors of the vehicle, and a processor of the vehicle. The one or more first sensors have a first modality and are configured to receive a first input from a passenger of the vehicle that is related to a request. The processor is configured to at least facilitate providing an instruction to the passenger for providing an additional input related to the request within a predetermined amount of time. The one or more second sensors have a second modality that is different from the first modality and are configured to receive a second input from the passenger that is related to the request. The processor is further configured to at least facilitate: interpreting the second input; and performing a vehicle action corresponding to the request based on the interpretation of the second input.

[0014] Further in example embodiments, the processor is further configured to at least facilitate determining the predetermined amount of time via adaptive learning based on a prior history of the passenger.

[0015] Further in example embodiments, the first input includes a voice command from the passenger; and the one or more first sensors include one or more microphones configured to receive the voice command from the passenger.

[0016] Further in example embodiments, the instruction includes an audio instruction; and the system further includes a speaker configured to provide the instruction.

[0017] Further in example embodiments, the instruction includes a visual instruction; and the system further includes a display screen configured to provide the instruction.

[0018] Further in example embodiments, the instructions inform the passenger to engage a particular input device in a particular directional manner within a predetermined amount of time based at least in part on the passenger's proximity to the particular input device; and the one or more second sensors include one or more input sensors configured to receive the second input with respect to engagement of the particular input device in the particular directional manner within the predetermined amount of time.

[0019] Further in example embodiments, the instructions inform the passenger to engage a particular input device typically used for a first vehicle function; and the second input is received via the one or more input sensors with respect to engagement of the particular input device for performing the request with respect to a second vehicle function different from and unrelated to the first vehicle function.

[0020] Further in example embodiments, the instructions inform the passenger to perform a particular gesture within a predetermined amount of time that is unrelated to any input device of the vehicle; and the one or more second sensors include one or more cameras configured to receive the second input with respect to the particular gesture within the predetermined amount of time.

[0021] Further in example embodiments, the instructions inform the passenger to swipe a steering wheel of the vehicle via a hand or finger of the passenger within a predetermined amount of time; and the second input is received via the one or more cameras with respect to the swipe of the steering wheel of the vehicle via the hand or finger of the passenger within the predetermined amount of time.

[0022] Further in example embodiments, the system is configured to be used by the passenger to request a plurality of different vehicle actions including opening and closing a window, adjusting a distance threshold for a cruise control, adjusting a volume of sound for a navigation system of the vehicle, and adjusting a zoom of a display of the navigation system.

[0023] In another example embodiment, a vehicle is provided that includes a body, a microphone, a processor, and one or more additional sensors. The microphone is disposed within the body and is configured to receive a first input from a passenger of the vehicle related to a request of the passenger, the first input including a verbal command of the passenger. The processor is configured to facilitate at least providing instructions to the passenger for providing an additional input related to the request within a predetermined amount of time. The one or more additional sensors have a different sensor modality than the microphone, the one or more additional sensors configured to receive a second input from the passenger related to the request within the predetermined amount of time in response to the instructions, the second input received via an input device engaged by the passenger. The processor is further configured to facilitate at least interpreting the second input and performing a vehicle action corresponding to the request based on the interpretation of the second input, wherein the vehicle action is different from a vehicle action typically used by the input device. BRIEF DESCRIPTION OF DRAWINGS

[0024] The present disclosure will be described below with reference to the following figures, wherein the same numerals denote the same elements, and wherein:

[0025] Figure 1 This is a functional block diagram of a vehicle including a control system for interacting with passengers of the vehicle, according to an exemplary embodiment.

[0026] Figure 2 It is an embodiment for interacting with passengers in a vehicle and can be combined with Figure 1 A flowchart of the process implemented by the vehicle (including its control system);

[0027] Figure 3 and Figure 4 Depicting according to exemplary embodiments Figure 2 An exemplary illustration of a sub-process of the process, which corresponds to the interaction with the passenger that leads to the use of a timer in the process;

[0028] Figure 5 Exemplary illustrations depict steps related to the use of a timer in process 200 according to an exemplary embodiment; and

[0029] Figure 6 An exemplary illustration depicts steps related to the active learning of a process according to an exemplary embodiment of process 200. Detailed Implementation

[0030] The following detailed description is merely exemplary in nature and is not intended to limit this disclosure or its application and use. Furthermore, it is not intended to be bound by the foregoing background information or any theories set forth in the following detailed description.

[0031] Figure 1 The figure illustrates a vehicle 100 according to an exemplary embodiment. As described in further detail below, according to an exemplary embodiment, vehicle 100 includes a control system 102 for interaction between a user and one or more passengers of the vehicle via a virtual assistant, as well as other components. Figure 1 as well as Figure 2 Process 200 and Figures 3-6 The implementation is described in further detail. In various embodiments, the control system 102 uses time-triggered manual input as part of the virtual assistant (when receiving, interpreting, and fulfilling passenger requests).

[0032] In various embodiments, vehicle 100 includes automobiles, such as any of several different types of automobiles, such as, for example, sedans, vans, trucks, sport utility vehicles (SUVs), etc. In some embodiments, vehicle 100 may also include motorcycles or other vehicles, such as aircraft, spacecraft, ships, etc., and / or one or more other types of mobile platforms (e.g., robots and / or another mobile platform).

[0033] In the depicted embodiment, vehicle 100 includes a body 104 disposed on a chassis 116. The body 104 substantially surrounds the other components of vehicle 100. The body 104 and chassis 116 may together form a frame. Vehicle 100 also includes a plurality of wheels 112. Each wheel 112 is rotatably coupled to chassis 116 near a corresponding corner of body 104 to facilitate movement of vehicle 100. In one embodiment, vehicle 100 includes four wheels 112, although this may vary in other embodiments (e.g., for trucks, motorcycles, and certain other vehicles).

[0034] The drive system 110 is mounted on the chassis 116 and drives the wheels 112, for example, via axle 114. In some embodiments, the drive system 110 includes a propulsion system having a motor 113 (e.g., in various embodiments, it includes one or more internal combustion engines, electric motors, etc.).

[0035] like Figure 1 As depicted, in various embodiments, the vehicle also includes a braking system 106 and a steering system 108. In an exemplary embodiment, the braking system 106 uses braking components to control the braking of the vehicle 100, which are controlled via input provided by the driver (e.g., via brake pedal 107) and / or automatically controlled via a control system (such as control system 102 and / or one or more other control systems).

[0036] Furthermore, in an exemplary embodiment, the steering system 108 controls the steering of the vehicle 100 via a steering component, which is controlled by input provided by the driver (e.g., via the steering wheel 109) and / or automatically controlled via a control system (such as control system 102 and / or one or more other control systems).

[0037] exist Figure 1 In the depicted embodiments, the control system 102 is coupled to the braking system 106, the steering system 108, and the drive system 110, and controls their operation and function. Furthermore, in various embodiments, the control system 102 is configured according to... Figure 2 The process described in 200 and Figures 3-6 The implementation method and, as described below in conjunction with its further description, provides interaction with one or more passengers of the vehicle via a virtual assistant.

[0038] Further as Figure 1 As depicted, in various embodiments, the control system 102 includes a sensor array 120, a display 130, and a controller 140, as described in greater detail below.

[0039] In various embodiments, the sensor array 120 includes various sensors that obtain sensor data regarding input from one or more passengers of the vehicle 100 (e.g., the driver and / or one or more other passengers of the vehicle 100). In the depicted embodiment, the sensor array 120 includes one or more input sensors 122, a microphone 124, and a camera 126. In certain embodiments, the sensor array 120 can also include one or more other sensors (e.g., regarding receiving other input, and / or obtaining various operating parameters, environmental conditions, etc.).

[0040] In various embodiments, the microphone 124 obtains audible input from one or more passengers of the vehicle 100, including words spoken by the passengers. Further in various embodiments, the camera 126 is configured to obtain visual input from one or more passengers of the vehicle 100, including hand or finger gestures and / or other movements by the passengers. In various embodiments, each of the input sensors 122, the microphone 124, and the camera 126 are disposed within the cabin of the vehicle 100, and obtain sensor data regarding input from the driver and other passengers from within the cabin of the vehicle 100.

[0041] In various embodiments, the display 130 provides information and instructions, as well as other content, to passengers of the vehicle 100 (including, in various embodiments, the driver as well as other passengers of the vehicle 100). As Figure 1 As depicted, in various embodiments, in addition to a visual (or video) component 134 (including one or more display screens) for displaying visual instructions, as well as other information and content, to passengers, the display 130 includes an audio component 132 (including one or more speakers) for displaying audio instructions, as well as other information and content, to passengers. In certain embodiments, the display 130 can also include, among other possibilities, a display screen, or a heads-up display, or a projector that projects images on an article, and / or in other embodiments, light or light surrounding a button, knob, or other input device, such as by flashing, rotating, and / or indicating to the user which button, etc.; and / or one or more other types of devices for providing indications, such as one or more haptic indications (e.g., rotating a steering wheel), and / or flashing light and / or buttons, etc.

[0042] In various embodiments, the controller 140 is coupled to the sensor array 120 and the display 130. Further, in various embodiments, the controller 140 receives sensor data from the sensor array 120, interprets and processes the sensor data, and provides instructions and other information and content via the display 130 based thereon. Further in various embodiments, the controller 140 controls various vehicle actions (e.g., including braking, steering, vehicle movement, cruise control settings, vehicle movement and operation, window operation, and the provision of navigation and other audiovisual information and content, including based on input obtained from passengers and interpretations and determinations made therefrom). In various embodiments, the controller 140 is also coupled to and controls the operation of the braking system 106, the steering system 108, and the drive system 110, as well as various other vehicle components (e.g., including a navigation system and other components not depicted).

[0043] In various embodiments, the controller 140 provides these functions in accordance with the steps of the process 200 depicted in Figure 2 and described in further detail below in connection with Figures 3-6 implementations (also described in further detail below).

[0044] As Figure 1 depicted, in various embodiments, the controller 140 includes a computer system (also referred to herein as the computer system 140), and includes a processor 142, a memory 144, an interface 146, a storage device 148, and a computer bus 150.

[0045] The processor 142 performs the computational and control functions of the controller 140, and can include any type of processor or multiple processors, a single integrated circuit such as a microprocessor, or any suitable number of integrated circuit devices and / or circuit boards working in cooperation to perform the functions of a processing unit. During operation, the processor 142 executes one or more programs 152 contained in the memory 144, and thus controls the general operation of the controller 140 and the general operation of the computer system of the controller 140 (typically in conjunction with operating system 154 and other programs 152, such as Figure 2 the process 200 of Figures 3-6 and implementations thereof, and as described further below in connection therewith).

[0046] The memory 144 can be any suitable type of memory, including various types of non-transitory computer readable storage media. In certain examples, the memory 144 is located and / or co-located on the same computer chip as the processor 142. In the depicted embodiment, the memory 144 stores the programs 152 described above, as well as stored values 157 (e.g., lookup tables, threshold values, and / or other values related to the process 200).

[0047] Interface 146 allows communication, for example, from system drives and / or another computer system to the computer system of controller 140, and can be implemented using any suitable method and apparatus. In one embodiment, interface 146 obtains various data from sensor array 120, as well as other possible data sources. Interface 146 can include one or more network interfaces to communicate with other systems or components. Interface 146 can also include one or more network interfaces to communicate with a technician, and / or one or more storage interfaces to connect to storage devices, such as storage device 148.

[0048] Storage device 148 can be any suitable type of storage device, including various different types of direct access storage and / or other memory devices. In one example embodiment, storage device 148 includes a program product from which memory 144 can receive program 152 that performs one or more embodiments of one or more processes of the present disclosure, such as Figure 2 the steps of process 200 of Figures 3-6 and implementations thereof, and as further described in connection therewith below. In another example embodiment, the program product can be stored directly in memory 144 and / or disk (e.g., disk 156) and / or otherwise accessed by memory 144 and / or disk, such as referenced hereinafter.

[0049] Bus 150 is used to transmit programs, data, status, and other information or signals between the various components of the computer system of controller 140. Bus 150 can be any suitable physical or logical means of connecting computer systems and components. This includes, but is not limited to, direct hard-wired connections, fiber optics, infrared, and wireless bus technology. During operation, program 152 is stored in memory 144 and executed by processor 142.

[0050] It will be appreciated that, although this example embodiment is described in the context of a fully functional computer system, those skilled in the art will recognize that the mechanisms of the present disclosure are capable of being distributed as a program product in one or more types of non-transitory computer-readable signal-bearing media having stored thereon the program and instructions and executed by a computer processor, such as processor 142, to perform and run the program.

[0051] Figure 2 is a flowchart of a process 200 for interacting with a passenger of a vehicle in accordance with example embodiments. In various embodiments, process 200 can be implemented in connection with Figure 1 vehicle 100 (including its control system 102) of Figures 3-6To further describe the process, Figures 3-6 An exemplary diagram depicts certain steps of process 200.

[0052] like Figure 2 As depicted, in various embodiments, process 200 begins (step 202) when the virtual assistant created for the vehicle is active. In various embodiments, this may include default features of vehicle 100, and / or may be via... Figure 1 One or more user input sensors 122 determine this through user input, etc. As used throughout this application, the term "passenger" may refer to the driver of vehicle 100 and / or one or more other passengers of vehicle 100.

[0053] In various embodiments, sensor data is acquired (step 204). Specifically, in some embodiments, the sensor data is obtained from... Figure 1 The sensor array 120 obtains, including user input from one or more passengers of vehicle 100 (e.g., via...). Figure 1 The sensor array 120 includes an input sensor 122, a microphone 124, and a camera 126.

[0054] In various embodiments, one or more first inputs are determined (step 206). The first inputs include initial instructions from the passenger, i.e., the passenger has instructions to travel via... Figure 1 The request is implemented by the control system 102. In some embodiments, the first input includes via... Figure 1 One or more microphones 124 capture verbal input from the passenger (e.g., “zoom in / out navigation display”, “roll down window”, “change cruise control settings”, etc.). In various embodiments, the nature of the first input is determined by a processor (such as...). Figure 1 The processor 142 determines this based on sensor data. In various embodiments, the "first input" may also include some initial interpretation by the processor 142 (and / or via a remote server), such as some natural language understanding. As described above, in some embodiments, the request relates to a request from one or more passengers of vehicle 100, wherein the request is initiated by a passenger. Alternatively, in some embodiments, the request may be initiated by vehicle 100 itself and / or via one or more of its devices and / or systems. For example, in some embodiments, vehicle 100's systems may proactively offer suggestions and / or other requests to the user (e.g., due to certain events or other triggers not originating from a user's request), and may also include suggestions as feedback using certain buttons / knobs (and / or other devices) in the system.

[0055] Furthermore, in various embodiments, a context is determined (step 208). In various embodiments, the context includes additional information relating to the passenger's request. In various embodiments, the context may include the location of the passenger making the request, such as location relative to the structure of vehicle 100 (e.g., driver's seat, front passenger seat, second row position such as left, middle, or right, or third row, etc.), and / or location relative to one or more input devices and / or relative to steering wheel 109, etc. Additionally, in some embodiments, the context may also include values ​​of one or more vehicle parameters, states, and / or conditions that may be related to the request (e.g., whether cruise control for vehicle 100 is currently active, whether the windows of vehicle 100 are currently up or down, etc.). In various embodiments, the context is determined via a processor (such as...). Figure 1 The processor 142) is determined based on sensor data.

[0056] In various embodiments, a strategy is selected (step 210). In various embodiments, the processor (such as...) Figure 1 The processor 142) determines an optimal strategy for soliciting further input from the passenger based on sensor data, the first input, and context. In various embodiments, the strategy includes selected means for the passenger to provide further input regarding the request. For example, in some embodiments, the strategy may include the user engaging a specific input device (e.g., possibly near the passenger), and / or causing the passenger to make a specific gesture or tap or otherwise contact a specific part or device of vehicle 100 (such as the steering wheel), etc. In various embodiments, the strategy is selected based on the type of request and the passenger's location (including the passenger's proximity to one or more input devices, other parts or devices of vehicle 100, etc.). Furthermore, in some embodiments, the strategy may also involve fulfilling a request from a passenger in a specific seating position (e.g., a passenger in the left rear of vehicle 100 may request increased audio volume for their seat and / or for a specific audio area near the seat, etc.). In one such embodiment, the strategy may also allow the passenger to control the volume (or other vehicle features) using window raise / lower buttons (buttons for opening / closing windows). In this example, any button or other control device of vehicle 100 (e.g., that can be pressed up / down and / or rotated, etc.) can be effectively used as a multi-controller to control other aspects of vehicle functions.

[0057] In various embodiments, instructions are provided to the passenger (step 212). In various embodiments, the processor (such as...) Figure 1 The processor 142) provides instructions to the passenger according to the strategy of step 210. In various embodiments, the instructions inform the passenger about how the passenger should provide additional and more specific input regarding the request.

[0058] In some embodiments, the additional input relates to a range or extent of a series of actions having a range of possible outcomes, such as zooming in or out of a navigation or other display, opening or closing a window, increasing or decreasing the audio for infotainment in vehicle 100, changing one or more cruise control settings, and so on. Furthermore, in some embodiments, the instruction requires the passenger to engage a specific input device in a specific directional manner (e.g., clockwise or counterclockwise rotation of a knob), which is transmitted via... Figure 1 One or more input sensors 122 detect. In some other embodiments, the instruction requires the passenger to make one or more gestures and / or other movements (e.g., such as raising an arm in a specific direction, tapping or stroking the steering wheel or other equipment or position of the vehicle 100 a predetermined number of times, etc.), which are transmitted via Figure 1 Camera 126 detects. In various embodiments, various possible instructions (e.g., input device-based or gesture-based) are based on strategies as determined in step 208, such as the type of request, the passenger's location, and proximity to the input device and / or other devices or locations of vehicle 100, etc.

[0059] In various embodiments, according to instructions provided by processor 142, during step 212 via Figure 1 The display 130 (e.g., a screen, or a head-up display, or a projector that projects images onto an object, and / or in other embodiments, light or light around it that controls a button, knob, or other input device, such as by flashing, rotating, and / or indicating to the user which button to use) provides instructions. In some embodiments, an audio description of the instructions is provided via an audio component 132 of the display 130. In some other embodiments, a visual description of the instructions is provided via a visual component 134 of the display 130. In some embodiments, one or more other indications may be provided, such as one or more tactile indications (e.g., rotating a steering wheel), and / or flashing light and / or buttons, and / or one or more other indications (e.g., those described above regarding the display).

[0060] In various embodiments, a timer is initiated (step 214). In various embodiments, the timer corresponds to a predetermined, finite amount of time provided to the passenger to respond to an instruction. Accordingly, in various embodiments, when the passenger responds to an instruction within this predetermined amount of time (e.g., by making a specified gesture, engaging a knob, tapping or stroking the steering wheel, etc.), the processor 142 recognizes this as a response to an instruction, rather than an unintentional action. In various embodiments, the predetermined amount of time may be stored... Figure 1The value 157 is stored in the memory 144. Furthermore, in various embodiments, the predetermined time amount can vary and can be customized for different passengers based on previous history, for example, as described below. Figure 6 Further description.

[0061] In various embodiments, one or more second inputs from the passenger are received via a sensor of a different modality than the sensor receiving the first input (e.g., in some embodiments, a different voice sensor or microphone than the one used to receive the first input). Specifically, in various embodiments, one or more additional sensors of sensor array 120 are used to obtain sensor data regarding additional inputs (also referred to as "second inputs" 216) provided by the passenger in response to an instruction. For example, in some implementations where the second input involves the passenger engaging an input device (such as a knob), the sensor data regarding the second input may be received via… Figure 1 The second input is obtained from one or more input sensors 122. Conversely, in some other implementations where the second input relates to the passenger's posture and / or tapping or touching one or more other devices or locations on the vehicle 100 (such as swiping the steering wheel), the sensor data regarding the second input can be obtained via... Figure 1 Acquired by one or more cameras 126.

[0062] In various embodiments, the second input is interpreted (step 218). Specifically, in various embodiments, the second input is interpreted via... Figure 1 The processor 142 interprets the information to more accurately determine the passenger's request. For example, in various embodiments, the interpretation of the second input may include the degree to which the passenger requests to zoom in or out of the display, or to increase or decrease the volume of the infotainment system, or to open or close a window, or to change one or more cruise control thresholds or settings, and so on.

[0063] In various embodiments, one or more actions are performed (step 220). Specifically, in some embodiments, Figure 1The processor 142 provides instructions to one or more other vehicle systems to perform a desired action (e.g., as described above in conjunction with step 218), which is then implemented via these vehicle systems. In various embodiments, the action is determined and instructed by one or more processors, based on input provided by a user using one or more input devices, which are utilized in a manner different from their usual use, based on temporary control provided to the user by the processor for this purpose. For example, in some embodiments, any available knobs, buttons, and / or other input devices may be utilized via temporary control provided to the user for one or more functions that are normally unrelated to that input device (e.g., by providing the user with temporary control over settings of one or more other systems). For example, in some embodiments, buttons typically used to open / close windows may be temporarily used to adjust the volume; and / or buttons on the rear seats typically used for air direction may be used to control channels and / or other entertainment display options on a television, and other variations in different embodiments, etc.

[0064] Furthermore, in various embodiments, adaptive learning is performed (step 222). In various embodiments, adaptive learning is performed via... Figure 1 The processor 142 executes the process, which considers one or more habits, preferences, etc., of the passenger, including, for example, how long a particular passenger typically takes to provide the second input for step 216, etc. In various embodiments, adaptive learning is used to update the threshold of the timer for step 214, and other possible uses of adaptive learning. In various embodiments, the resulting value (e.g., regarding the timer threshold) is stored in... Figure 1 The value 157 is stored in memory 144. An illustration of an exemplary implementation of the adaptive learning in step 222 is shown in... Figure 6 As described herein, and further described below under its conditions. In some embodiments, learning may be performed relative to one or more users and / or relative to one or more other vehicles, etc.

[0065] In various embodiments, process 200 then terminates at step 224.

[0066] refer to Figure 3 Provided according to exemplary embodiments Figure 2 An exemplary illustration of sub-process 207 of process 200. Furthermore, in the exemplary embodiment, sub-process 207 corresponds to an interaction with the passenger that results in a timer being used during the process (e.g., corresponding to...). Figure 2 Steps 206-216). According to an exemplary embodiment, Figure 3The illustration corresponds to an implementation in which a passenger is prompted to provide a second input via an input device (e.g., in one embodiment, a knob that is turned clockwise or counterclockwise).

[0067] like Figure 3 As depicted, in an exemplary embodiment, one or more first inputs 206(1) are received from the driver 301 (e.g., corresponding to...). Figure 2 The first set of inputs in step 206 (e.g., "Please lower the navigation voice"). In an exemplary embodiment, the request is further indicated via the voice interface 302 through one or more additional first inputs 206(2) (including commands (e.g., for lowering the navigation volume)). Figure 3 As depicted, in some embodiments, 212(1) represents a conceptual or logical representation of an instruction, while 212(2) represents a specific representation of the concept. Furthermore, as... Figure 3 As depicted, additional instructions are provided to the driver 301 (e.g., using statements such as, “To lower the navigation volume, please turn the knob counterclockwise now,” or using instructions from the virtual assistant to first reduce the navigation volume based on deterministic / logical factors, and to notify the user that “the volume can be further reduced or increased using the rotatable knob,” etc.).

[0068] In various embodiments, additional input 206(3) is received from the user (e.g., user engagement of a knob, and, for example, references 212(3)...212(4) and the start and end of a timer 306 and 307) according to the dialogue manager (or display) 303. In various embodiments, along a certain time frame t1-tn, the user can rotate the knob or other device, and events regarding the angle of the knob (or other device) can occur and be detected, and subsequently one or more functions (e.g., navigation volume) can be updated based on this. Alternatively, in some other embodiments, the system may also provide the user with further input to improve user input. For example, in one embodiment, if the user is using a gesture of swiping on the steering wheel, the system may tell him to make a larger gesture (so that it will be better detected) or a smaller gesture, or an indication that it will terminate the interaction (e.g., "knob returned to normal use"), and so on.

[0069] refer to Figure 4 Provided according to exemplary embodiments Figure 2 Another exemplary illustration of subprocess 207 of process 200. According to an exemplary embodiment, Figure 4 The illustration corresponds to an implementation in which a passenger is prompted to provide a second input (e.g., via one or more gestures of the passenger) when there is no input device.

[0070] like Figure 4As depicted, in an exemplary embodiment, one or more first inputs 206(1) are received from the driver 401 (e.g., corresponding to...). Figure 2 The first set of inputs in step 206 (e.g., "zoom out"). In an exemplary embodiment, the request is further indicated via voice interface 402 through one or more additional first inputs 206(2) (including commands (e.g., for zooming out the display)). Figure 4 As depicted, in some embodiments, a first instruction 212(1) is provided to the voice interface 402 (e.g., a first input 206(1) corresponds to an initial broad indication from the passenger regarding a request (e.g., providing an instruction for narrowing down)). Furthermore, as... Figure 4 As depicted, additional instruction 212(2) is provided to driver 401 (e.g., using a statement such as: "To narrow down, now move the steering wheel to the left").

[0071] In various embodiments, according to the dialogue manager (or display) 403, additional input 206(3) is received from the user (e.g., Figure 1 (Activation of 126 and / or other sensors). Furthermore, in various embodiments, a timer is utilized (corresponding to...). Figure 2 Step 214 includes starting the timer 406 and ending the timer 407.

[0072] Figure 5 The steps related to the use of a timer in process 200 according to an exemplary embodiment are described (e.g., corresponding to...). Figure 2 An exemplary illustration of step 214). Figure 5 As depicted, in an exemplary embodiment, user voice command 502 (e.g., corresponding to...) Figure 2 The first input 206 is received between the start period 504 and the end period 506, followed by system dialogue 508. In various embodiments, as part of the timer, the passenger's response at point "A" 510 represents early manual input after a prompt (e.g., instruction), while the passenger's response at point "B" 512 represents late manual input after a prompt. Conversely, also in various embodiments: the passenger's response at point "C" 514 represents very late manual input after a prompt, while the passenger's response at point "D" 516 represents very early input before a prompt. In various embodiments, the processor 142 customizes and adjusts the timer accordingly based on the history of the passenger's actions, for example, as Figure 2part of the learning of step 222. For example, in one embodiment, after a user has used the system and knows that he or she can change the volume using the knob (or other device), upon the user asking to set the volume by voice, the user can say "decrease the volume" and simultaneously begin to rotate the knob (as he or she already knows that the system will change the volume based on the knob). In an exemplary embodiment, in such a case, the learning (e.g., the system needs to monitor the knob early on) allows the system to avoid missing the manual input.

[0073] Figure 6 depicts an exemplary illustration 600 of steps of process 200 (e.g., corresponding to Figure 2 step 222) related to active learning of the process, in accordance with exemplary embodiments, in which instructions for completing the request are provided to the passenger via one or more gestures captured by one or more cameras. As Figure 6 shown, in various embodiments, speech recognition is provided (corresponding to step 206), as well as a dialog manager (corresponding to step 210), text-to-speech (corresponding to step 212), and gesture detection (e.g., relative finger / hand detection, corresponding to step 216). Further in various embodiments, a timer is triggered (corresponding to step 214), and updated displays are provided as appropriate (corresponding to step 220). In various embodiments, internal camera images 602 are obtained (e.g., in some embodiments, also corresponding to the second input of step 216). In various embodiments, learning is performed via a history of the passenger's actions, such as in customizing the timer for a particular passenger, and other possible customizations (e.g., in some embodiments, customizing the strategy and / or instructions based on passenger preferences, etc.).

[0074] Accordingly, in accordance with exemplary embodiments, methods, systems, and vehicles are provided for interacting with one or more passengers of a vehicle via a virtual assistant. In various embodiments, time-triggered manual inputs are used as part of the virtual assistant in receiving, interpreting, and implementing a passenger's request of the vehicle. As described above, in various embodiments, the user provides an initial input (e.g., via a voice command), followed by additional input of a different modality or type (e.g., via engagement of an input device such as a knob, captured by one or more input sensors, or via one or more gestures captured by one or more cameras), which is based on a strategy designed via a computer processor and provided to the passenger in the form of instructions, which are then implemented via the user upon providing the additional input.

[0075] It will be appreciated that the systems, vehicles, and methods can differ from those depicted in the drawings and described herein. For example, in embodiments different from those Figure 1 depicted in the drawings and / or described above in connection with Figure 1 the embodiments, in various embodiments, the system, vehicle, and / or method can include one or more of the following features: Figure 1The vehicle 100 (including the control system 102 and / or other components thereof) can vary. Similarly, it will be understood that the steps of the process 200 and their implementation can differ from those depicted in Figures 2-6 and / or the various steps of the process 200 can occur simultaneously and / or in a different order than depicted in Figures 2-6 and / or described above in connection therewith.

[0076] While at least one example embodiment has been presented in the foregoing detailed description, it should be appreciated that a vast number of variations exist. It should also be appreciated that the example embodiment or embodiments are only examples, and are not intended to limit the scope, applicability or configuration of the disclosure in any way. Rather, the foregoing detailed description will provide those skilled in the art with a convenient road map for implementing an example embodiment or embodiments. It should be understood that various changes can be made in the function and arrangement of elements without departing from the scope of the disclosure as set forth in the appended claims and the legal equivalents thereof.

Claims

1. A method comprising: The vehicle receives a first input related to a request from a passenger of the vehicle via one or more first sensors, the one or more first sensors having a first mode; The vehicle's processor provides the passenger with instructions to provide additional input related to the request within a predetermined time period; In response to the instruction, the vehicle receives a second input from the passenger in relation to the request within a predetermined time period via one or more second sensors, the one or more second sensors having a second mode different from the first mode; The processor interprets the second input; as well as The processor performs vehicle actions corresponding to the request based on the interpretation of the second input.

2. The method of claim 1, wherein the predetermined time amount is determined by the processor via adaptive learning based on previous history.

3. The method of claim 1, wherein the first input comprises a voice command from the passenger and is received via one or more microphones of the vehicle.

4. The method of claim 3, wherein the instructions include audio instructions provided via a speaker of the vehicle coupled to the processor.

5. The method of claim 3, wherein the instructions include visual instructions provided via a display screen of the vehicle coupled to the processor.

6. The method according to claim 3, wherein: The instruction is based at least in part on the proximity of the passenger to the specific input device to notify the passenger to engage the specific input device in a specific direction and manner within the predetermined time period; as well as The second input is received via one or more input sensors about engagement with the specific input device in the specific orientation manner within the predetermined time period.

7. The method according to claim 6, wherein: The instruction instructs the passenger to engage the specific input device typically used for a first vehicle function; and The second input is received via the one or more input sensors with respect to the engagement of the input device, for performing the request with respect to a second vehicle function that is different from and unrelated to the first vehicle function.

8. The method according to claim 3, wherein: The instruction instructs the passenger to perform a specific gesture, independent of any input device of the vehicle, within the predetermined time period; and The second input is received via one or more cameras with respect to the specific pose within the predetermined time period.

9. The method according to claim 8, wherein: The instruction instructs the passenger to move the vehicle's steering wheel with their hand or finger within the predetermined time period; as well as The second input is received via the one or more cameras with respect to the swiping of the steering wheel of the vehicle by the passenger's hand or fingers within the predetermined time period.

10. A system comprising: One or more first sensors of the vehicle, the one or more first sensors being configured to receive a request-related first input from a passenger of the vehicle, the one or more first sensors having a first mode; The vehicle's processor is configured to at least facilitate providing the passenger with instructions for providing additional input related to the request within a predetermined time period; as well as One or more second sensors of the vehicle, the one or more second sensors being configured to receive second input from the passenger in relation to the request within the predetermined time period in response to the instruction, the one or more second sensors having a second mode different from the first mode; The processor is also configured to at least facilitate: Explain the second input; and The vehicle action corresponding to the request is performed based on the interpretation of the second input.

Images