Interactive system and method

Abstract

A method, computer program product, and computing system for: monitoring an interior environment of a motor vehicle; determining a personal state of an occupant within the interior environment of the motor vehicle; and interacting with the occupant based at least in part on the occupant's personal state.

Description

Technical Field

[0001] The present invention relates to interactive systems and, more particularly, to interactive systems for use within a motor vehicle. Background Art

[0002] The capabilities of today's consumer electronic devices are constantly expanding. And when these consumer electronic devices are coupled with systems that capture audio (such as microphones, etc.) and systems that capture video (such as cameras, etc.), these consumer electronic devices can allow users to use their voice, face and / or eyes as a "password" to grant them access to these consumer electronic devices, for example.

[0003] Furthermore, by using such supplemental information obtained via, for example, microphones and cameras, the way in which such consumer electronic devices function can be altered / manipulated based on what such consumer electronic devices "hear" and "see". Summary of the invention

[0004] In one implementation, a computer-implemented method is executed on a computing device and includes: monitoring an interior environment of a motor vehicle; determining a personal state of an occupant within the interior environment of the motor vehicle; and interacting with the occupant based at least in part on the occupant's personal state.

[0005] One or more of the following features may be included. The occupant may be the driver of the motor vehicle. The personal state of the occupant within the interior environment of the motor vehicle may include one or more of the following: the physiological / physical / external state of the occupant within the interior environment of the motor vehicle; and the psychological / emotional / internal state of the occupant within the interior environment of the motor vehicle. Interacting with the occupant based at least in part on the personal state of the occupant may include: physically changing the interior environment of the motor vehicle. Physically changing the interior environment of the motor vehicle may include one or more of the following: adjusting the temperature of the interior environment of the motor vehicle; adjusting one or more windows of the motor vehicle; adjusting a sunroof of the motor vehicle; adjusting one or more interior lights of the motor vehicle; adjusting an entertainment system of the motor vehicle; adjusting a seating system of the motor vehicle; and adjusting a ventilation system of a motor vehicle. Interacting with the occupant based at least in part on the occupant's personal state may include one or more of: triggering an audible alarm; triggering a visual alarm; physically engaging an occupant within the interior environment of the motor vehicle; visually engaging an occupant within the interior environment of the motor vehicle; and audibly engaging an occupant within the interior environment of the motor vehicle. Interacting with the occupant based at least in part on the occupant's personal state may include adjusting the manner in which a voice-based communication system engages an occupant within the interior environment of the motor vehicle. Adjusting the manner in which the voice-based communication system attracts occupants within the interior environment of the motor vehicle may include one or more of the following: adjusting the speed of speech used by the voice-based communication system to attract occupants within the interior environment of the motor vehicle; adjusting the vocabulary of speech used by the voice-based communication system to attract occupants within the interior environment of the motor vehicle; adjusting the amount of speech used by the voice-based communication system to attract occupants within the interior environment of the motor vehicle; adjusting the intonation of speech used by the voice-based communication system to attract occupants within the interior environment of the motor vehicle; and adjusting the timbre of speech used by the voice-based communication system to attract occupants within the interior environment of the motor vehicle.

[0006] In another implementation, a computer program product resides on a computer-readable medium and stores a plurality of instructions. The plurality of instructions, when executed by a processor, causes the processor to perform operations including: monitoring an interior environment of a motor vehicle; determining a personal state of an occupant within the interior environment of the motor vehicle; and interacting with the occupant based at least in part on the personal state of the occupant.

[0007] One or more of the following features may be included. The occupant may be the driver of the motor vehicle. The personal state of the occupant within the interior environment of the motor vehicle may include one or more of the following: the physiological / physical / external state of the occupant within the interior environment of the motor vehicle; and the psychological / emotional / internal state of the occupant within the interior environment of the motor vehicle. Interacting with the occupant based at least in part on the personal state of the occupant may include: physically changing the interior environment of the motor vehicle. Physically changing the interior environment of the motor vehicle may include one or more of the following: adjusting the temperature of the interior environment of the motor vehicle; adjusting one or more windows of the motor vehicle; adjusting a sunroof of the motor vehicle; adjusting one or more interior lights of the motor vehicle; adjusting an entertainment system of the motor vehicle; adjusting a seating system of the motor vehicle; and adjusting a ventilation system of a motor vehicle. Interacting with the occupant based at least in part on the occupant's personal state may include one or more of: triggering an audible alarm; triggering a visual alarm; physically engaging an occupant within the interior environment of the motor vehicle; visually engaging an occupant within the interior environment of the motor vehicle; and audibly engaging an occupant within the interior environment of the motor vehicle. Interacting with the occupant based at least in part on the occupant's personal state may include adjusting the manner in which a voice-based communication system engages an occupant within the interior environment of the motor vehicle. Adjusting the manner in which the voice-based communication system attracts occupants within the interior environment of the motor vehicle may include one or more of the following: adjusting the speed of speech used by the voice-based communication system to attract occupants within the interior environment of the motor vehicle; adjusting the vocabulary of speech used by the voice-based communication system to attract occupants within the interior environment of the motor vehicle; adjusting the amount of speech used by the voice-based communication system to attract occupants within the interior environment of the motor vehicle; adjusting the intonation of speech used by the voice-based communication system to attract occupants within the interior environment of the motor vehicle; and adjusting the timbre of speech used by the voice-based communication system to attract occupants within the interior environment of the motor vehicle.

[0008] In another implementation, a computing system includes a processor and a memory configured to perform operations including: monitoring an interior environment of a motor vehicle; determining a personal state of an occupant within the interior environment of the motor vehicle; and interacting with the occupant based at least in part on the occupant's personal state.

[0009] One or more of the following features may be included. The occupant may be the driver of the motor vehicle. The personal state of the occupant within the interior environment of the motor vehicle may include one or more of the following: the physiological / physical / external state of the occupant within the interior environment of the motor vehicle; and the psychological / emotional / internal state of the occupant within the interior environment of the motor vehicle. Interacting with the occupant based at least in part on the personal state of the occupant may include: physically changing the interior environment of the motor vehicle. Physically changing the interior environment of the motor vehicle may include one or more of the following: adjusting the temperature of the interior environment of the motor vehicle; adjusting one or more windows of the motor vehicle; adjusting a sunroof of the motor vehicle; adjusting one or more interior lights of the motor vehicle; adjusting an entertainment system of the motor vehicle; adjusting a seating system of the motor vehicle; and adjusting a ventilation system of a motor vehicle. Interacting with the occupant based at least in part on the occupant's personal state may include one or more of: triggering an audible alarm; triggering a visual alarm; physically engaging an occupant within the interior environment of the motor vehicle; visually engaging an occupant within the interior environment of the motor vehicle; and audibly engaging an occupant within the interior environment of the motor vehicle. Interacting with the occupant based at least in part on the occupant's personal state may include adjusting the manner in which a voice-based communication system engages an occupant within the interior environment of the motor vehicle. Adjusting the manner in which the voice-based communication system attracts occupants within the interior environment of the motor vehicle may include one or more of the following: adjusting the speed of speech used by the voice-based communication system to attract occupants within the interior environment of the motor vehicle; adjusting the vocabulary of speech used by the voice-based communication system to attract occupants within the interior environment of the motor vehicle; adjusting the amount of speech used by the voice-based communication system to attract occupants within the interior environment of the motor vehicle; adjusting the intonation of speech used by the voice-based communication system to attract occupants within the interior environment of the motor vehicle; and adjusting the timbre of speech used by the voice-based communication system to attract occupants within the interior environment of the motor vehicle.

[0010] The details of one or more implementations are set forth in the accompanying drawings and the description below. Other features and advantages will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 is a schematic diagram of a consumer electronic device that performs interactive processing according to an embodiment of the present invention;

[0012] Figure 2 It is incorporated into the execution Figure 1 A schematic diagram of a motor vehicle for interacting with a consumer electronic device; and

[0013] Figure 3 According to an embodiment of the present invention Figure 1 Flowchart of the interactive processing.

[0014] Like reference numbers in the various drawings represent like elements. DETAILED DESCRIPTION

[0015] System Overview

[0016] refer to Figure 1 , illustrating an interaction process 10. The interaction process 10 may reside on and may be executed by a consumer electronic device 12. Examples of consumer electronic devices 12 may include, but are not limited to, vehicle electronic systems. Examples of such vehicle electronic systems may include any type of infotainment / electronic system incorporated into a motor vehicle, such as a vehicle navigation system, a vehicle music system, a vehicle video system, a vehicle telephone system, a vehicle climate control system, and a vehicle control system, etc.

[0017] The instruction sets and subroutines of the interactive process 10 may be executed by one or more processors (not shown) and one or more memory architectures (not shown) included in the consumer electronic device 12, wherein these instruction sets and subroutines may be stored on a storage device 14 coupled to the consumer electronic device 12. Examples of the storage device 14 may include, but are not limited to: a hard drive; a random access memory (RAM); a read-only memory (ROM); and all forms of flash memory devices. The consumer electronic device 12 may execute an operating system, examples of which may include, but are not limited to, Microsoft Windows TM 、Android TM 、iOS TM , Linux TM Or a custom operating system.

[0018] When configured as a vehicle infotainment system, consumer electronic device 12 may be configured to perform a variety of different functions that may be of interest / useful to a user (e.g., user 16). Examples of such functions may include, but are not limited to: radio functions (e.g., enabling the playback of terrestrial and satellite radio stations); audio functions (e.g., enabling the playback of audio, which may be disk-based or stored locally on storage device 14); video functions (e.g., enabling the playback of video, which may be disk-based or stored locally on storage device 14); phone functions (e.g., enabling the placement and receipt of phone calls); navigation functions (e.g., enabling the performance of navigation / guidance functions); and communication functions (e.g., enabling the sending and receiving of emails / text messages / instant messages).

[0019] When configured as a vehicle infotainment system, the consumer electronics device 12 may include a plurality of buttons (e.g., physical buttons or electronic buttons) that enable the above functions to be selected. For example, the above radio function may be selected via a "radio" button 18; the above audio function may be selected via an "audio" button 20; the above video function may be selected via a "video" button 22; the above phone function may be selected via a "phone" button 24; the above navigation function may be selected via a "navigation" button 26; and the above communication function may be selected via a "communication" button 28.

[0020] When configured as a vehicle infotainment system, the consumer electronic device 12 may be configured to connect to one or more external systems (e.g., external system 30). Examples of external system 30 may include, but are not limited to: a cellular phone; a smart phone; a tablet computing device; a portable computing device; and a handheld entertainment device (e.g., such as a gaming device, etc.). When connected to the consumer electronic device 12, the external system 30 may be releasably coupled to the consumer electronic device 12 via a hardwired connection (e.g., a USB cable 32). Optionally, the external system 30 may be wirelessly coupled to the consumer electronic device 12 via a wireless communication channel 34 established between the external system 30 and an antenna 36 of the consumer electronic device 12. Examples of the wireless communication channel 34 may include, but are not limited to, a Bluetooth communication channel. As is known in the art, Bluetooth is a telecommunications industry specification that allows, for example, mobile phones, computers, and personal digital assistants to interconnect using a short-range wireless connection.

[0021] Consumer electronics device 12 and / or external system 30 may be configured to wirelessly couple to / access an external network (eg, network 38). Examples of network 38 may include, but are not limited to, the Internet, a cellular network, a WiFi network, and / or a cloud-based computing platform.

[0022] As discussed above, the consumer electronic device 12 may be configured to perform a variety of different functions that may be of interest to / useful to a user (e.g., user 16). Some of these functions may reside locally on (provided by) the consumer electronic device 12. Additionally / optionally, some of these functions may reside remotely on (provided by) the external system 30. Examples of such remotely resident functions may include telephone functions (e.g., enabling the use of the external system 30 to make and receive calls via the consumer electronic device 12) and communication functions (enabling the user 16 to send / receive emails, send / receive text messages, and / or send / receive instant messages via the consumer electronic device 12 using the external system 30). The consumer electronic device 12 may also include a display screen 40 and one or more knobs / dials 42, 44 for enabling the use of these functions.

[0023] The consumer electronic device 12 may include a microphone assembly 46 and a voice processing system 48 (such as those available from Nuance Communications, Inc. of Burlington, Massachusetts). Thus, the consumer electronic device 12 may be configured to accept verbal commands (e.g., verbal commands 50) spoken and provided by (in this example) the user 16. These verbal commands (e.g., verbal commands 50) may be configured to allow the user 16 to access and control the above-mentioned functions in a hands-free manner. The consumer electronic device 12 may also include a camera assembly 52, which may be configured to capture video 54 of (in this example) the user 16, wherein the captured video 54 may be processed by the video processing system 56. The consumer electronic device 12 may also include a speaker assembly 58, which may be configured to present audio 60 to (in this example) the user 16, wherein the audio 60 may be generated by the consumer electronic device 12 and / or the external system 30.

[0024] As discussed above, the interaction process 10 may reside on and be executed by a consumer electronic device 12, examples of which may include, but are not limited to, a vehicle electronic system. Figures 2 to 3 , the consumer electronic device 12 may be located within (or part of) a motor vehicle 100, wherein the vehicle 100 may include one or more occupants. For this particular example, the vehicle 100 is shown to include two occupants, namely the user 16 (who is driving the vehicle 100) and the passenger 102.

[0025] Interaction Processing

[0026] The interaction process 10 may monitor 200 an interior environment 104 (e.g., a passage compartment) of the motor vehicle 100 and may determine 202 a personal state of an occupant (e.g., a user 16 or a passenger 102) within the interior environment 104 of the motor vehicle 100. Upon determining 202 the personal state, the interaction process 10 may interact 204 with the occupant (e.g., a user 16 or a passenger 102) based at least in part on the personal state of the occupant (e.g., a user 16 or a passenger 102).

[0027] Examples of personal states 202 determined for an occupant (e.g., user 16 or passenger 102) within interior environment 104 of motor vehicle 100 may include, but are not limited to, one or more of the following:

[0028] The physiological / physical / external state of the occupant (e.g., user 16 or passenger 102) within the interior environment 104 of the motor vehicle 100. For example, is the occupant (e.g., user 16 or passenger 102) in a hurry? Is the occupant (e.g., user 16 or passenger 102) sleepy? Is the occupant (e.g., user 16 or passenger 102) impaired? Is the occupant (e.g., user 16 or passenger 102) injured? Is the occupant (e.g., user 16 or passenger 102) sick? Is the occupant (e.g., user 16 or passenger 102) experiencing a medical emergency?

[0029] The psychological / emotional / internal state of the occupant (e.g., user 16 or passenger 102) within the interior environment 104 of the motor vehicle 100. For example, is the occupant (e.g., user 16 or passenger 102) nervous? Is the occupant (e.g., user 16 or passenger 102) angry? Is the occupant (e.g., user 16 or passenger 102) cheerful? Is the occupant (e.g., user 16 or passenger 102) talkative?

[0030] Additionally, the interaction process 10 may use various methods to determine 202 a personal state (eg, physiological / physical / external state, psychological / emotional / internal state) of an occupant (eg, user 16 or passenger 102 ) within the interior environment 104 of the motor vehicle 100 .

[0031] Audio indication of personal status

[0032] Thus, the interaction process 10 can utilize the microphone assembly 46 and the speech processing system 48 to monitor words / phrases spoken by an occupant of the motor vehicle 100 (e.g., the user 16 or the passenger 102) to determine 202 the personal state (e.g., physiological / physical / external state, psychological / emotional / internal state) of the occupant (e.g., the user 16 or the passenger 102).

[0033] For example:

[0034] • If the occupant (eg, user 16 or passenger 102) is speaking faster, this may indicate that the occupant (eg, user 16 or passenger 102) is hurried / rushed / alert.

[0035] • If the occupant (eg, user 16 or passenger 102) is speaking at a normal speed, this may indicate that the occupant (eg, user 16 or passenger 102) is relaxed / calm.

[0036] • If the occupant (eg, user 16 or passenger 102) is speaking more slowly, this may indicate that the occupant (eg, user 16 or passenger 102) is sleepy / tired.

[0037] • If the occupant (eg, user 16 or passenger 102) is snoring / silent, this may indicate that the occupant (eg, user 16 or passenger 102) is asleep.

[0038] • If the occupant (eg, user 16 or passenger 102) is talking / yelling loudly, this may indicate that the occupant (eg, user 16 or passenger 102) is angry / stressed.

[0039] • If the occupant (eg, user 16 or passenger 102) is being verbose, this may indicate that the occupant (eg, user 16 or passenger 102) is pleasant / chatty.

[0040] • If the occupant (eg, user 16 or passenger 102) slurs their speech, this may indicate that the occupant (eg, user 16 or passenger 102) is impaired.

[0041] • If the occupant (eg, user 16 or passenger 102) is using a lot of ascending inflections, this may indicate that the occupant (eg, user 16 or passenger 102) is uncertain / confused.

[0042] Video indication of personal status

[0043] In addition, the interaction process 10 can utilize the camera component 52 and the video processing system 56 to visually monitor the occupants of the motor vehicle 100 (e.g., the user 16 or the passenger 102) to determine 202 personal state (e.g., physiological / physical / external state, psychological / emotional / internal state).

[0044] For example:

[0045] If the occupant (e.g., user 16 or passenger 102) has a hyper-alert facial expression (e.g., fully open eyes, active head movement), this may indicate that the occupant (e.g., user 16 or passenger 102) is hurried / hurried / alert.

[0046] • If the occupant (eg, user 16 or passenger 102) has a normal facial expression (eg, normally open eyes, typical head movement), this may indicate that the occupant (eg, user 16 or passenger 102) is relaxed / calm.

[0047] If the occupant (eg, user 16 or passenger 102) has a tired facial expression (eg, partially closed eyes, wobbly head motion), this may indicate that the occupant (eg, user 16 or passenger 102) is sleepy / tired.

[0048] • If the occupant (eg, user 16 or passenger 102) has a sleeping facial expression (eg, fully closed eyes, tilted / tilted head), this may indicate that the occupant (eg, user 16 or passenger 102) is asleep.

[0049] If the occupant (eg, user 16 or passenger 102) has an anxious facial expression (eg, frowning, squinting eyebrows, jerky head movements), this may indicate that the occupant (eg, user 16 or passenger 102) is irritated / stressed.

[0050] • If the occupant (eg, user 16 or passenger 102) has a pleasant facial expression (eg, a smile with cheerful head movement), this may indicate that the occupant (eg, user 16 or passenger 102) is pleasant / chatty.

[0051] • If the occupant (eg, user 16 or passenger 102) has an impaired facial expression (eg, partially closed eyes, dilated pupils), this may indicate that the occupant (eg, user 16 or passenger 102) is impaired.

[0052] • If the occupant (eg, user 16 or passenger 102) has a confused facial expression (eg, eyes opened to varying degrees, head tilted), this may indicate that the occupant (eg, user 16 or passenger 102) is unsure / confused.

[0053] As discussed above, the interaction process 10 may utilize audio information and / or video information to determine 202 a personal state (e.g., physiological / physical / external state, psychological / emotional / internal state) of an occupant (e.g., user 16 or passenger 102) of the motor vehicle 100. Once the personal state (e.g., physiological / physical / external state, psychological / emotional / internal state) is determined 202, the interaction process 10 may interact 204 with the occupant (e.g., user 16 or passenger 102) based at least in part on the personal state of the occupant (e.g., user 16 or passenger 102).

[0054] When interacting 204 with an occupant (e.g., user 16 or passenger 102) based at least in part on the occupant's (e.g., user 16 or passenger 102) personal state (e.g., physiological / physical / external state, psychological / emotional / internal state), the interaction processing 10 can perform various functions, examples of which may include, but are not limited to, adjusting the manner in which a voice-based communication system (e.g., voice processing system 48) engages an occupant (e.g., user 16 or passenger 102) within the interior environment 104 of the motor vehicle 100.

[0055] When adjusting 206 the manner in which a voice-based communication system (e.g., voice processing system 48) engages an occupant (e.g., user 16 or passenger 102) within the interior environment 104 of the motor vehicle 100, the interaction processing 10 can customize the interaction with the occupant (e.g., user 16 or passenger 102) to accommodate or influence the personal state (e.g., physiological / physical / external state, psychological / emotional / internal state) of the occupant (e.g., user 16 or passenger 102).

[0056] For example:

[0057] The interaction process 10 may adjust 208 the speed of speech used by the speech-based communication system (e.g., speech processing system 48) to engage an occupant (e.g., user 16 or passenger 102) within the interior environment 104 of the motor vehicle 100. For example, if the personal state of the occupant (e.g., user 16 or passenger 102) within the interior environment 104 of the motor vehicle 100 indicates that they are in a hurry, a faster speed may be used. Conversely, if the personal state of the occupant (e.g., user 16 or passenger 102) within the interior environment 104 of the motor vehicle 100 indicates that they are relaxed, a slower speed may be used.

[0058] The interaction process 10 may adjust 210 the vocabulary of speech used by the speech-based communication system (e.g., speech processing system 48) in order to appeal to an occupant (e.g., user 16 or passenger 102) within the interior environment 104 of the motor vehicle 100. For example, if the personal state of an occupant (e.g., user 16 or passenger 102) within the interior environment 104 of the motor vehicle 100 indicates that they are nervous, a more formal or neutral vocabulary may be used. Conversely, if the personal state of an occupant (e.g., user 16 or passenger 102) within the interior environment 104 of the motor vehicle 100 indicates that they are relaxed, a more informal or colloquial vocabulary may be used.

[0059] The interaction process 10 may adjust 212 the amount of speech that the speech-based communication system (e.g., speech processing system 48) uses in order to engage an occupant (e.g., user 16 or passenger 102) within the interior environment 104 of the motor vehicle 100. For example, if the personal status of the occupant (e.g., user 16 or passenger 102) within the interior environment 104 of the motor vehicle 100 indicates that they are in a hurry, fewer words / shorter sentences may be used. Conversely, if the personal status of the occupant (e.g., user 16 or passenger 102) within the interior environment 104 of the motor vehicle 100 indicates that they are talkative, more words / longer sentences may be used.

[0060] The interaction process 10 may adjust 214 the inflection of speech used by the speech-based communication system (e.g., speech processing system 48) in order to appeal to an occupant (e.g., user 16 or passenger 102) within the interior environment 104 of the motor vehicle 100. For example, if the personal status of an occupant (e.g., user 16 or passenger 102) within the interior environment 104 of the motor vehicle 100 indicates that they are annoyed, a more monotone / calm inflection may be used. Conversely, if the personal status of an occupant (e.g., user 16 or passenger 102) within the interior environment 104 of the motor vehicle 100 indicates that they are cheerful, a more varied / lively inflection may be used.

[0061] The interaction process 10 may adjust 216 the timbre of speech used by the speech-based communication system (e.g., speech processing system 48) in order to appeal to an occupant (e.g., user 16 or passenger 102) within the interior environment 104 of the motor vehicle 100. For example, if the personal state of an occupant (e.g., user 16 or passenger 102) within the interior environment 104 of the motor vehicle 100 indicates that they are cheerful, speech spoken with a bright and tight timbre may be used. Conversely, if the personal state of an occupant (e.g., user 16 or passenger 102) within the interior environment 104 of the motor vehicle 100 indicates that they are sad, speech spoken with a somber and breathy timbre may be used.

[0062] As discussed above, when interacting 204 with an occupant (e.g., user 16 or passenger 102) based at least in part on the occupant's (e.g., user 16 or passenger 102) personal state (e.g., physiological / physical / external state, psychological / emotional / internal state), the interaction process 10 can perform various functions, another example of which can include, but is not limited to, physically changing 218 the interior environment 104 of the motor vehicle 100.

[0063] For example, in the event that it is determined that the occupant (e.g., user 16) who is driving the motor vehicle 100 appears to be impaired and / or drowsy, the interaction process 10 may take remedial action and may physically change 218 the interior environment 104 of the motor vehicle 100 in an attempt to make the occupant (e.g., user 16) more alert. One or more of the remedial actions described below may be: a) automatically initiated by the interaction process 10; or b) provided to the occupant (e.g., user 16) by the interaction process 10 and initiated only with the approval of the occupant (e.g., user 16).

[0064] For example:

[0065] The interaction process 10 may adjust 220 the temperature of the interior environment 104 of the motor vehicle 100. For example, in the event that an occupant (e.g., user 16) appears impaired and / or drowsy, the interaction process 10 may provide a control signal 62 to the interface system 64, which may be configured to interact with a climate control system (not shown) of the motor vehicle 100 such that, for example, the temperature within the interior environment 104 may be reduced in an attempt to increase the alertness level of the occupant (e.g., user 16).

[0066] The interaction process 10 may adjust 222 one or more windows of the motor vehicle 100. For example, in the event that an occupant (e.g., user 16) appears impaired and / or drowsy, the interaction process 10 may provide a control signal 62 to the interface system 64, which may be configured to interact with a window control system (not shown) of the motor vehicle 100 so that, for example, one or more windows (not shown) of the motor vehicle 100 may be lowered in an attempt to increase the occupant's (e.g., user 16) alertness level.

[0067] The interaction process 10 may adjust 224 a sunroof of the motor vehicle 100. For example, in the event that the occupant (e.g., user 16) appears impaired and / or drowsy, the interaction process 10 may provide a control signal 62 to the interface system 64, wherein the interface system 64 may be configured to interact with a sunroof control system (not shown) of the motor vehicle 100 such that, for example, a sunroof (not shown) of the motor vehicle 100 may be opened in an attempt to increase the occupant's (e.g., user 16) alertness level.

[0068] The interaction process 10 may adjust 226 one or more interior lights of the motor vehicle 100. For example, in the event that an occupant (e.g., user 16) appears impaired and / or drowsy, the interaction process 10 may provide a control signal 62 to the interface system 64, which may be configured to interact with a lighting control system (not shown) such that, for example, the interior environment 104 may be brightened in an attempt to increase the occupant's (e.g., user 16) alertness level.

[0069] The interaction process 10 may adjust 228 the entertainment system of the motor vehicle 100. For example, in the event that the occupant (e.g., user 16) appears to be impaired and / or drowsy, the interaction process 10 may provide a control signal 62 to the interface system 64, wherein the interface system 64 may be configured to interact with the entertainment system (e.g., consumer electronic device 12) within the motor vehicle 100 so that, for example, loud music may be played. Alternatively, a type of music (e.g., classical music / rock music / heavy metal music) that the occupant (e.g., user 16) finds energizing (based on personal preferences) may be played. Also optionally, the occupant (e.g., user 16) of the motor vehicle 100 may be asked to participate in a synthesis conversation (e.g., the interaction process 10 and the occupant (e.g., user 16) may participate in a conversation involving speech recognition, natural language understanding (NLU), dialog management, natural language generation (NLG), and speech synthesis) in an attempt to increase the occupant's (e.g., user 16) alertness level.

[0070] The interaction process 10 may adjust 230 a seating system of the motor vehicle 100. For example, in the event that the occupant (e.g., user 16) appears impaired and / or drowsy, the interaction process 10 may provide a control signal 62 to an interface system 64, wherein the interface system 64 may be configured to interact with a seating system (not shown) of the motor vehicle 100 such that, for example, the driver's seat may be adjusted, for example, to place the occupant (e.g., user 16) in a more upright driving position in an attempt to increase the occupant's (e.g., user 16) alertness level. To configure the interaction process 10 in this manner, active approval of the occupant (e.g., user 16) may be required due to the intrusive nature of such an interaction.

[0071] The interaction process 10 may adjust 232 the ventilation system 100 of the motor vehicle. For example, in the event that the occupant (e.g., user 16) appears impaired and / or drowsy, the interaction process 10 may provide a control signal 62 to the interface system 64, wherein the interface system 64 may be configured to interact with the ventilation system (not shown) of the motor vehicle 100 such that, for example, a high velocity airflow may be directed to the occupant (e.g., user 16) in an attempt to increase the occupant's (e.g., user 16) alertness level.

[0072] Again and as discussed above, when interacting 204 with an occupant (e.g., user 16 or passenger 102) based at least in part on the occupant's (e.g., user 16 or passenger 102) personal state (e.g., physiological / physical / external state, psychological / emotional / internal state), the interaction process 10 can perform various functions. One or more of the functions described below can be: a) automatically initiated by the interaction process 10; or b) provided to the occupant (e.g., user 16) by the interaction process 10 and initiated only with the approval of the occupant (e.g., user 16). Examples of such functions can include, but are not limited to:

[0073] The interaction process 10 triggers 234 an audible alarm. For example, in the event that the occupant (e.g., user 16) appears impaired and / or drowsy, the interaction process 10 may present an audible alarm (via the speaker assembly 58) in an attempt to increase the occupant's (e.g., user 16) alertness level.

[0074] The interaction process 10 triggers 236 a visual alert. For example, in the event that the occupant (e.g., user 16) appears impaired and / or drowsy, the interaction process 10 may present a visual alert (via the display screen 40 and / or an interior lighting system (not shown)) in an attempt to increase the occupant's (e.g., user 16) alertness level.

[0075] The interaction process 10 physically engages 238 an occupant (e.g., user 16) within the interior environment 104 of the motor vehicle 100. For example, in the event that the occupant (e.g., user 16) appears impaired and / or drowsy, the interaction process 10 may physically engage the occupant (e.g., user 16) via, for example, a vibration motor in a steering wheel (not shown), a vibration motor in a seat (not shown), and / or a water mist assembly (not shown) in an attempt to increase the occupant's (e.g., user 16) alertness level. To configure the interaction process 10 in this manner, active approval by the occupant (e.g., user 16) may be required due to the intrusive nature of such an interaction.

[0076] The interaction process 10 visually engages 240 an occupant (e.g., user 16) within the interior environment 104 of the motor vehicle 100. For example, in the event that the occupant (e.g., user 16) appears impaired and / or sleepy, the interaction process 10 may visually engage the occupant (e.g., user 16) via, for example, the display screen 40 in an attempt to increase the occupant's (e.g., user 16) alertness level, wherein the occupant (e.g., user 16) may be required to touch the display screen 40 at defined intervals to avoid triggering an alarm. To configure the interaction process 10 in this manner, active approval by the occupant (e.g., user 16) may be required due to the intrusive nature of such an interaction.

[0077] The interaction process 10 audibly engages 242 an occupant (e.g., user 16) within the interior environment 104 of the motor vehicle 100. For example, in a situation where the occupant (e.g., user 16) appears impaired and / or drowsy, the interaction process 10 may audibly engage the occupant (e.g., user 16) via, for example, the speaker assembly 58 and the microphone assembly 48 in an attempt to increase the occupant's (e.g., user 16) alertness level, wherein the occupant (e.g., user 16) may utilize a speech-based communication system (e.g., speech processing system 48) to engage the occupant (e.g., user 16) in a synthesized conversation (e.g., the interaction process 10 and the occupant (e.g., user 16) may engage in a conversation involving speech recognition, natural language understanding (NLU), dialog management, natural language generation (NLG), and speech synthesis) to avoid triggering an alarm.

[0078] General

[0079] As will be appreciated by those skilled in the art, the present invention may be embodied as a method, system or computer program product. Thus, the present invention may take the form of an all-hardware embodiment, an all-software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining software aspects and hardware aspects, wherein the software aspects and hardware aspects may be generally referred to herein as "circuits," "modules," or "systems." Furthermore, the present invention may take the form of a computer program product on a computer-usable storage medium having a computer-usable program code embodied in the medium.

[0080] Any suitable computer-usable or computer-readable medium may be utilized. A computer-usable or computer-readable medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, apparatus, or propagation medium. More specific examples of computer-readable media (a non-exhaustive list) may include the following: an electrical connection with one or more wires, a portable computer floppy disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a transmission medium (such as a transmission medium supporting the Internet or an intranet), or a magnetic storage device. A computer-usable or computer-readable medium may also be paper or other suitable medium on which the program is printed, because the program may be captured electronically via, for example, optical scanning of paper or other media, and then compiled, interpreted, or otherwise processed in an appropriate manner if necessary, and then stored in a computer memory. In the context of this document, a computer-usable or computer-readable medium may be any medium that may contain, store, communicate, propagate, or transmit a program for use by or in connection with an instruction execution system, device, or apparatus. The computer usable medium may include a propagated data signal with the computer usable program code embodied therein, either in baseband or as part of a carrier wave. The computer usable program code may be transmitted using any appropriate medium (including but not limited to the Internet, wired, optical fiber cable, RF, etc.).

[0081] The computer program code for performing the operation of the present invention can be written in an object-oriented programming language such as Java, Smalltalk, C++, etc. However, the computer program code for performing the operation of the present invention can also be written in a traditional procedural programming language such as the "C" programming language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer, partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer can be connected to the user's computer via a local area network / wide area network / Internet (e.g., network 14).

[0082] The present invention is described with reference to the flowchart and / or block diagram of the method, device (system) and computer program product according to an embodiment of the present invention. It should be understood that each frame in the flowchart and / or block diagram and the combination of frames in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer / special-purpose computer / other programmable data processing device, so that the instructions executed by the processor of the computer or other programmable data processing device create components for implementing the functions / actions specified in the frames of the flowchart and / or block diagram.

[0083] These computer program instructions may also be stored in a computer-readable memory, which may instruct a computer or other programmable data processing device to operate in a specific manner so that the instructions stored in the computer-readable memory produce a product including an instruction portion for implementing the functions / actions specified in the boxes of the flowchart and / or block diagram.

[0084] Computer program instructions may also be loaded onto a computer or other programmable data processing device to perform a series of operational steps on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions / actions specified in the boxes of the flowchart and / or block diagram.

[0085] The flow chart and block diagram in the figure can illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention.In this regard, each frame in the flow chart or block diagram can represent the module, segment or part of the code, which includes one or more executable instructions for realizing the specified logical function.It should also be noted that in some optional implementations, the function mentioned in the frame may not conform to the order mentioned in the figure.For example, the two frames shown in sequence may actually be executed substantially at the same time, or these frames may sometimes be executed in reverse order according to the functions involved.It should also be noted that the combination of each frame in the block diagram and / or the flow chart and the frame in the block diagram and / or the flow chart can be realized by a hardware-based special-purpose system or a combination of special-purpose hardware and computer instructions that performs a specified function or action.

[0086] The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. As used herein, the singular forms "a", "an", and "the" are also intended to include the plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "comprises / comprising" when used in this specification specifies the presence of the features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0087] The corresponding structures, materials, actions and equivalents of all parts or steps plus function elements in the appended claims are intended to include any structure, material or action for performing functions in combination with other claimed elements specifically claimed. The description of the present invention is provided for the purpose of illustration and description, but is not intended to be exhaustive or limited to the disclosure in the disclosed form. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the present invention. The embodiments are selected and described to best explain the principles and practical applications of the present invention and to enable those of ordinary skill in the art to understand the various embodiments of the present invention and the various modifications suitable for the intended specific use.

[0088] Having thus described the disclosure of the present application in detail, and by reference to embodiments thereof, it will be apparent that modifications and variations are possible without departing from the scope of the disclosure as defined in the appended claims.

Claims

1. A computer-implemented method, executed on a computing device, comprising: Using an infotainment system with a microphone and a camera to monitor the interior environment of a mobile vehicle; determining a personal state of an occupant within the interior environment of the motor vehicle based on information provided by the microphone and the camera; interacting with the occupant based at least in part on the occupant's personal status, wherein using the infotainment system comprises: monitoring the interior environment using the camera and the microphone, and wherein the interacting comprises performing at least one of the following actions: visually engaging the occupant and requiring the occupant to touch a display screen at defined intervals, and engaging the occupant in a synthetic conversation using a speech-based communication system; In response to the occupant not completing the action, triggering an alarm; and In response to the occupant having completed the action, triggering the alarm is avoided.

2. The computer-implemented method of claim 1 , wherein: Interacting with the occupant based at least in part on the occupant's personal state includes adjusting one or more windows of the motor vehicle.

3. The computer-implemented method of claim 1 , wherein: Interacting with the occupant based at least in part on the occupant's personal state includes adjusting the manner in which a voice-based communication system engages the occupant within the interior environment of the motor vehicle.

4. The computer-implemented method of claim 1 , wherein: Interacting with the occupant based at least in part on the occupant's personal state includes adjusting a rate of speech used by a voice-based communication system to engage an occupant within the interior environment of the motor vehicle, wherein the extent of the adjustment depends on the personal state.

5. The computer-implemented method of claim 1 , wherein: Interacting with the occupant based at least in part on the occupant's personal state includes adjusting a vocabulary of speech used by a voice-based communication system to engage an occupant within the interior environment of the motor vehicle, wherein the vocabulary is adjusted at least in part based on the occupant's personal state.

6. The computer-implemented method of claim 1 , wherein: Interacting with the occupant based at least in part on the occupant's personal state includes adjusting an amount of speech used by a voice-based communication system to engage the occupant within the interior environment of the motor vehicle.

7. The computer-implemented method of claim 1 , wherein: Interacting with the occupant based at least in part on the occupant's personal status includes adjusting inflection of speech used by a voice-based communication system communicating with the occupant.

8. The computer-implemented method of claim 1 , wherein: Interacting with the occupant based at least in part on the occupant's personal status includes adjusting a timbre of speech used by a voice-based communication system.

9. The computer-implemented method of claim 1 , wherein: Monitoring the interior environment of the motor vehicle includes providing speech from the occupant using the microphone to a speech processing system configured for natural language understanding.

10. The computer-implemented method of claim 1, wherein: Interacting with the occupant includes lowering a window of the motor vehicle.

11. The computer-implemented method of claim 1 , wherein: Interacting with the occupant also includes adjusting an entertainment system of the motor vehicle.

12. The computer-implemented method of claim 1, wherein: Interacting with the occupant also includes adjusting interior lights of the motor vehicle.

13. The computer-implemented method of claim 1, wherein: Interacting with the occupant also includes adjusting a seat in the motor vehicle.

14. The computer-implemented method of claim 1, wherein: The personal state includes at least one of an impairment state and a drowsy state.

15. An article comprising a tangible and non-transitory computer-readable medium having encoded thereon instructions that, when executed by a processor, cause the processor to perform operations comprising: monitoring an interior environment of a motor vehicle using an infotainment system having a microphone, a voice processing system, and a camera; determining a personal state of an occupant within the interior environment of the motor vehicle; and using the voice processing system to interact with the occupant based at least in part on the personal state of the occupant, wherein using the infotainment system comprises: The internal environment is monitored using the camera and the microphone, and wherein the interaction comprises performing at least one of the following actions: visually engaging the occupant and requiring the occupant to touch a display screen at defined intervals, and engaging the occupant in a synthetic conversation using a speech-based communication system; and The processor also performs: In response to the occupant not completing the action, triggering an alarm; and In response to the occupant having completed the action, triggering the alarm is avoided.

16. The article of claim 15, wherein: Interacting with the occupant based at least in part on the occupant's personal status includes adjusting a sunroof of the motor vehicle.

17. The article of claim 15, wherein: Interacting with the occupant based at least in part on the occupant's personal state includes adjusting the manner in which a voice-based communication system engages the occupant within the interior environment of the motor vehicle, wherein the voice-based communication system is configured to understand and generate natural language and synthesize speech.

18. The article of claim 17, wherein: The method of adjusting the voice-based communication system to attract occupants within the interior environment of the motor vehicle includes: adjusting the speed of the speech used by the voice-based communication system to attract occupants within the interior environment of the motor vehicle; adjusting the vocabulary of the speech used by the voice-based communication system to attract occupants within the interior environment of the motor vehicle; adjusting the amount of speech used by the voice-based communication system to attract occupants within the interior environment of the motor vehicle; adjusting the intonation of the speech used by the voice-based communication system to attract occupants within the interior environment of the motor vehicle; and adjusting the timbre of the speech used by the voice-based communication system to attract occupants within the interior environment of the motor vehicle.

19. A device configured to monitor the interior environment of a motor vehicle, the device comprising an infotainment system in the motor vehicle, wherein the infotainment system comprises a processor, a memory, a microphone, a camera and a voice processing system, the voice processing system is used to interact with an occupant, the microphone and the camera are arranged to monitor the interior environment of the motor vehicle and the personal status of the occupant, wherein, The interaction includes performing at least one of the following actions: visually engaging the occupant and requiring the occupant to touch a display screen at defined intervals, and engaging the occupant in a synthetic conversation using a speech-based communication system; as well as The processor is configured to: In response to the occupant not completing the action, triggering an alarm; and In response to the occupant having completed the action, triggering the alarm is avoided.

20. The apparatus of claim 19, wherein: Interacting with the occupant based at least in part on the occupant's personal state includes adjusting the manner in which a voice-based communication system engages the occupant within the interior environment of the motor vehicle.

21. The apparatus according to claim 20, wherein: Adjusting the manner in which the voice-based communication system attracts occupants within the interior environment of the motor vehicle includes one or more of the following: adjusting the speed of speech used by the voice-based communication system to attract occupants within the interior environment of the motor vehicle; adjusting the vocabulary of speech used by the voice-based communication system to attract occupants within the interior environment of the motor vehicle; adjusting the amount of speech used by the voice-based communication system to attract occupants within the interior environment of the motor vehicle; adjusting the intonation of speech used by the voice-based communication system to attract occupants within the interior environment of the motor vehicle; and adjusting the timbre of speech used by the voice-based communication system to attract occupants within the interior environment of the motor vehicle.

22. The apparatus of claim 19, wherein: The infotainment system is configured to wirelessly couple to a cloud-based computing platform.

23. The apparatus of claim 19, wherein: The infotainment system is configured to play terrestrial and satellite radio stations, play disk-based stored audio, play stored video, make and receive calls, provide navigation information, and send and receive messages, including email messages, text messages, and instant messages, and wherein the infotainment system is also configured to interface with external systems.

24. A computer program product storing instructions, which, when executed by a processor, causes the processor to implement the steps of any one of the methods of claims 1-14.

Images