Systems and methods for driver assistance

Abstract

Systems and methods are disclosed for driver assistance. In one example a method for an advanced driver alert system for a vehicle comprises monitoring, via a user-facing camera, facial movement of a user over time. The method includes determining a plurality of facial movement metrics based on the facial movement over an adjustable time window, transforming the plurality of facial movement metrics into a machine readable representation of the adjustable time window, and determining, via a code reading model, one or more user requests based on the machine readable representation of the facial movement metrics. The method includes executing one or more user assistance operations based on the determined user requests.

Description

SYSTEMS AND METHODS FOR DRIVER ASSISTANCEFIELD

[0001] The present invention generally relates to the field of driver assistance systems and, more particularly, to methods and systems for determining user requests based on facial movement and executing user assistance operations based on the determined user requests.BACKGROUND

[0002] Advanced driver-assistance systems (ADAS) are technologies that monitor vehicle operating conditions and assist with the safe operation of a vehicle. Through a human-machine interface. ADAS use automated technology’, such as sensors and cameras, to detect driving obstacles, driver errors, and other situations, and respond accordingly. Some examples of ADAS include sounding an alert when a driver error is indicated, assisting with driving tasks, and monitoring the driving environment for obstacles.

[0003] Further, some ADAS may include the vehicle in electronic communication with emergency assistance functions via an assistance button. For example, the assistance button may function as a signal to call for emergency assistance. However, in situations where a driver or passenger cannot access the assistance button, such as in an emergency situation or an accident, the driver may be unable to request emergency assistance. The lack of access to driver assistance tools may cause a delay in the driver obtaining a medical or emergency response. Further, in some situations, a driver may be unable to verbally communicate, or may which to request assistance discreetly, due the nature of the emergency.SUMMARY

[0004] Embodiments are disclosed for systems and methods for activating vehicle assistance. The systems and methods described herein use an advanced driver-assistance system (ADAS) to determine and command user requests based on monitoring facial movement of a vehicle occupant.

[0005] In one aspect, a method for an advanced driver alert system for a vehicle comprises monitoring, via a user-facing camera, facial movement of a user over time. The method includes determining a plurality of facial movement metrics based on the facial movement over an adjustable time window. The method includes transforming the plurality of facial movement metrics into a machine readable representation of the adjustable time window. One or more user requests is determined based on the machine readable representation of the facialmovement metrics and a code reading model. The method includes executing one or more user assistance operations based on the determined user requests.

[0006] In another aspect, a method for an advanced driver alert system for a vehicle comprises operating in a first monitoring mode in response to a first set of conditions and operating in a second monitoring mode in response to a second set of conditions. The first monitoring mode comprises monitoring, via a user-facing camera, eye movement of a user over time and predicting, via a psychophysiological state prediction model, one or more psychophysiological states of the user based on the eye movement. The second monitoring mode comprises monitoring, via the user-facing camera, at least one of eye movement and lip movement of the user over time, determining, via a code reading model, one or more user requests based on the at least one of eye movement and lip movement, and executing one or more user assistance operations based on to the user requests. In some embodiments of the method, the second set of conditions may include a psychophysiological state of the user greater than or equal to a first threshold, and the first set of conditions may include a psychophysiological state of the user less than the first threshold.

[0007] In yet another aspect, an advanced driver alert system for a vehicle comprises an infotainment system configured to execute a plurality of user assistance operations. The system includes a camera configured to capture a sequence of facial movements over time, a non- transitory memory storing instructions, a psychophysiological state prediction model, and a code reading model. A processor is communicably coupled to the camera, the infotainment system, and the non-transitory memory. When executing the instructions, the processor is configured to monitor in a first operating mode, via the camera, eye movement of a user over time and predict, via the psychophysiological state prediction model, one or more psychophysiological states of the user based on the eye movement. In response to an indication of greater than a first threshold psychophysiological state, the processor monitors in a second monitoring mode, via the camera, at least one of eye movement and lip movement of the user over time, and determines, via the code reading model, one or more user requests based on the at least one of eye movement and lip movement. The processor executes one or more user assistance operations based on the one or more determined user requests.

[0008] In this way, the ADAS determines and executes user requests for assistance based on facial movement. As a result, a driver in an unsafe situation, who is unable to physically access a driver assistance request button, or is differently abled, may request assistance without using their hands or generating audible speech. The approach may shift between two monitoring modes based on operating conditions, thereby providing efficient and discreetmethods for driver assistance without demanding more processing power than a system that monitors user facial movement in one mode only.

[0009] It should be understood that the summary above is provided to introduce in simplified form a selection of concepts that are further described in the detailed description. It is not meant to identify key or essential features of the claimed subject matter, the scope of which is defined uniquely by the claims that follow the detailed description. Furthermore, the claimed subject matter is not limited to implementations that solve any disadvantages noted above or in any part of this disclosure.BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The disclosure may be better understood from reading the following description of non-limiting embodiments, with reference to the attached drawings, wherein below:

[0011] FIG. 1 is a partial view of an exemplary interior of a cabin of a vehicle in accordance with one or more embodiments of the present disclosure;

[0012] FIG. 2 is a block diagram of an exemplary in-vehicle computing system of the vehicle of FIG. 1 in accordance with one or more embodiments of the present disclosure;

[0013] FIG. 3 is a schematic diagram illustrating of an example of a process for predicting psychophysiological states and determining user requests based on facial movement metrics of a user in accordance with one or more embodiments of the present disclosure;

[0014] FIG. 4 is a block diagram depicting an example of a facial movement interpretation device in accordance with one or more embodiments of the present disclosure;

[0015] FIG. 5 is an exemplary' advanced driver assistance system, including the vehicle of FIGS. 1-2 in accordance with one or more embodiments of the present disclosure;

[0016] FIG. 6 is a flowchart of an example of a method for operating in a first monitoring mode in response to a first set of conditions and operating in a second monitoring mode in response to a second set of conditions in accordance with one or more embodiments of the present disclosure;

[0017] FIG. 7 is a flowchart of an example of a method for operating in a first operating mode including predicting one or more psychophysiological states for a user based on machine readable representation of a plurality of second order eye movement metrics; and

[0018] FIG. 8 is a flowchart of an example of a method for operating in a second operating mode including determining one or more user requests based on machine readable representation of a plurality’ of facial movement metrics in accordance with one or more embodiments of the present disclosure.DETAILED DESCRIPTION

[0019] Examples will be provided below for illustration. The descriptions of the various examples will be presented for purposes of illustration, but are not intended to be exhaustive or limited to the embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments.

[0020] The present disclosure provides systems and methods for monitoring facial movements, via a user-facing camera, to detect requests for user assistance operations. The present disclosure includes methods and systems that capture facial movements, transforming the facial movements into a machine readable representation, and determining, via a code reading model, one or more user requests based on the machine readable representation of the facial movement metrics. The disclosed methods and systems facilitate the transformation of facial movement data into a machine readable representation within an adjustable time window. The machine readable representation is achieved through the determination of facial movement metrics, which include, but are not limited to. blink number, blink pattern, blink duration, lip shapes, lip patterns, and lip shape duration. The metrics may correspond to plurality of messages which may be communicated by a user for commanding one or more of a plurality7of user assistance operations. For example, the code reading model may be trained on eye movement data to detect one or more of a plurality of messages in Morse code corresponding to user requests, and lip movement data to detect one or more of a plurality of lip reading messages corresponding to user requests. The user requests determined via the code reading model may be output to an infotainment system that is configured to execute a plurality7of user assistance operations.

[0021] In some examples, the plurality of user assistance operations may include activating a vehicle function, ceasing a vehicle function, adjusting a vehicle operating condition, and requesting emergency assistance, which may be commanded by the user using one or more eye blinking or lip reading codes. For example, the vehicle occupant may use eye blinking to activate high beams, open / close doors or trunk, and activate / deactivate climate control or wipers, and so on. The disclosed systems and methods provide a hands-free and discreet request for vehicle assistance, yvhich may be helpful to a driver or a passenger yvho is unable to physically access an assistance request button or wishes to avoid detection of the request for assistance, for example, due to the nature of the emergency.

[0022] In some examples of the present disclosure, the systems and methods for monitoring facial movements to detect requests for user assistance operations may be included in an advanced driver assistance system (ADAS) that includes two or more monitoring modes. The present disclosure includes systems and methods for operating in a first monitoring mode in response to a first set of conditions and operating in a second monitoring mode in response to a second set of conditions. The first monitoring mode may include monitoring, via a userfacing camera, eye movement of a user over time and predicting, via a psychophysiological state prediction model, one or more psychophysiological states of the user based on the eye movement, such as drowsiness, cognitive load, stress, fatigue, and others. The second monitoring mode may include monitoring facial movements to detect requests for user assistance operations. The ADAS may include control logic that determines which mode of monitoring mode is commanded, such as shifting from the first monitoring mode to the second monitoring mode, or vice versa, in response to predicting greater than threshold stress, drowsiness, or other psychophysiological state of the user.

[0023] The technical advantage of the present disclosure lies in its ability to provide representation of facial movements that can be used by machine learning models to determine user requests and to predict various psychophysiological states. By incorporating logic that controls which monitoring mode is commanded, the approach may demand no more processing power than a system that monitors user facial movement in one mode only.

[0024] The approaches disclosed herein may be applicable in driver monitoring systems for vehicles such as the vehicle shown in FIG. 1. At least some of the user assistance operations may be executed by, or in communication with, an infotainment system, such as the infotainment system shown in FIG. 2. One example of a process for predicting psychophysiological states and determining user requests based on facial movement metrics of a user is depicted in FIG. 1. The process may be implemented by a facial movement interpretation device, as shown in FIG. 4, which is configured to analyze facial movement characteristics and derive second order facial movement metrics therefrom. Second order facial movement metrics may represent a plurality of messages corresponding to user requests, and additionally, or alternatively, may indicative of various psychophysiological states such as cognitive load or stress. The device and methods involved in this process are capable of transforming discrete sequences of facial movements into a machine readable representation that can be utilized for the determination of user requests or prediction of psychophysiological states, for example, depending on a mode of operation of the ADAS. An example of an ADAS including a facial movement interpretation device that is configured to shift between predictingpsychophysiological states of a user and determining user requests is shown in FIG. 5. One example of a method for controlling the ADAS to shift between operating in a first monitoring mode and operating in a second monitoring mode is shown in FIG. 6. The first monitoring mode may include monitoring eye movement of a user over time and predicting one or more psychophysiological states of the user based on the eye movement. The second monitoring mode may include monitoring facial movement of the user over time (e.g., at least one of eye movement and lip movement), determining one or more user requests based on the facial movement, and executing one or more user assistance operations based on to the user requests. The method for predicting one or more psychophysiological states for a user, based on machine readable representation of a plurality of second order eye movement metrics, is outlined in the flowchart of FIG. 7. The method for determining user requests, based on machine readable representation of a plurality of facial movement metrics, is outlined in the flowchart of FIG. 8.

[0025] FIG. 1 shows an example partial view of an interior of a cabin 100 of a vehicle 102, in which a driver and / or one or more passengers may be seated. Vehicle 102 of FIG. 1 may be a motor vehicle including drive wheels (not shown) and an internal combustion engine 104. Internal combustion engine 104 may include one or more combustion chambers which may receive intake air via an intake passage and exhaust combustion gases via an exhaust passage. Vehicle 102 may be a road automobile, among other ty pes of vehicles. In some examples, vehicle 102 may include a hybrid propulsion system including an energy conversion device operable to absorb energy from vehicle motion and / or the engine and convert the absorbed energy to an energy form suitable for storage by an energy storage device. Vehicle 102 may include a fully electric vehicle, incorporating fuel cells, solar energy capturing elements, and / or other energy storage systems for powering the vehicle.

[0026] As shown, an instrument panel 106 may include various displays and controls accessible to a human driver (also referred to as the user) of vehicle 102. For example, instrument panel 106 may7include a touch screen 108 of an in-vehicle computing system or infotainment system 109 (e.g., an infotainment system), an audio system control panel, and an instrument cluster 110. Touch screen 108 may receive user input to in-vehicle computing system or infotainment system 109 for controlling audio output, visual display output, user preferences, control parameter selection, and so on. While the example system shown in FIG. 1 includes audio system controls that may be performed via a user interface of in-vehicle computing system or infotainment system 109, such as touch screen 108 without a separate audio system control panel, in other embodiments, the vehicle may include an audio system control panel, which may include controls for a conventional vehicle audio system such as aradio, compact disc player, MP3 player, and so on. The audio system controls may include features for controlling one or more aspects of audio output via one or more speakers 112 of a vehicle speaker system. For example, the in-vehicle computing system or the audio system controls may control a volume of audio output, a distribution of sound among the individual speakers of the vehicle speaker system, an equalization of audio signals, and / or any other aspect of the audio output. In further examples, in-vehicle computing system or infotainment system 109 may adjust aradio station selection, a playlist selection, a source of audio input (e.g.. from radio or CD or MP3), and so on, based on user input received directly via touch screen 108, or based on data regarding the user (such as a physical state and / or environment of the user) received via one or more external devices 150 and / or a mobile device 128. The audio system of the vehicle may include an amplifier (not shown) coupled to plurality of loudspeakers (not shown). In some embodiments, one or more hardware elements of in-vehicle computing system or infotainment system 109, such as touch screen 108, a display screen 111, various control dials, knobs and buttons, memory, processor(s), and any interface elements (e.g., connectors or ports) may form an integrated head unit that is installed in instrument panel 106 of the vehicle. The head unit may be fixedly or removably attached in instrument panel 106. In additional or alternative embodiments, one or more hardware elements of in-vehicle computing system or infotainment system 109 may be modular and may be installed in multiple locations of the vehicle.

[0027] Cabin 100 may include one or more sensors for monitoring the vehicle, the user, and / or the environment. For example, cabin 100 may include one or more cameras configured to monitor one or more vehicle occupants and the cabin, one or more seat-mounted pressure sensors configured to measure the pressure applied to the seat to determine the presence of a user, door sensors configured to monitor door activity, humidity sensors to measure the humidity content of the cabin, microphones to receive user input in the form of voice commands, to enable a user to conduct telephone calls, and / or to measure ambient noise in cabin 100, and so on. It is to be understood that the above-described sensors and / or one or more additional or alternative sensors may be positioned in any suitable location of the vehicle. For example, sensors may be positioned in an engine compartment, on an external surface of the vehicle, and / or in other suitable locations for providing information regarding the operation of the vehicle, ambient conditions of the vehicle, a user of the vehicle, and so on. Information regarding ambient conditions of the vehicle, vehicle status, or vehicle driver may also be received from sensors external to / separate from the vehicle (that is, not part of the vehicle system), such as sensors coupled to external devices 150 and / or mobile device 128.

[0028] Cabin 100 may also include one or more user objects, such as mobile device 128, that are stored in the vehicle before, during, and / or after travelling. Mobile device 128 may include a smart phone, a tablet, a laptop computer, a portable media player, and / or any suitable mobile computing device. Mobile device 128 may be connected to in-vehicle computing system via a communication link 130. Communication link 130 may be wired (e.g., via Universal Serial Bus (USB), Mobile High-Definition Link (MHL), High-Definition Multimedia Interface (HDMI). Ethernet, and so on) or wireless (e.g., via Bluetooth®. Wi-Fi®. Wi-Fi Direct®, Near-Field Communication (NFC), cellular connectivity, and so on) and configured to provide two-way communication between the mobile device and the in-vehicle computing system. (Bluetooth® is a registered trademark of Bluetooth SIG, Inc., Kirkland, WA. Wi-Fi® and Wi-Fi Direct® are registered trademarks of Wi-Fi Alliance, Austin, Texas.) Mobile device 128 may include one or more wireless communication interfaces for connecting to one or more communication links (e.g., one or more of the example communication links described above). The wireless communication interface may include one or more physical devices, such as antenna(s) or port(s) coupled to data lines for carrying transmitted or received data, as well as one or more modules / drivers for operating the physical devices in accordance with other devices in the mobile device. For example, communication link 130 may provide sensor and / or control signals from various vehicle systems (such as vehicle audio system, climate control system, and so on) and touch screen 108 to mobile device 128 and may provide control and / or display signals from mobile device 128 to the in-vehicle systems and touch screen 108. Communication link 130 may also provide power to mobile device 128 from an in-vehicle power source in order to charge an internal battery of the mobile device.

[0029] In-vehicle computing system or infotainment system 109 may also be communicatively coupled to additional devices operated and / or accessed by the user but located external to vehicle 102, such as one or more external devices 150. In the depicted embodiment, external devices are located outside of vehicle 102 though it will be appreciated that in alternate embodiments, external devices may be located inside cabin 100. The external devices may include a server computing system, personal computing system, portable electronic device, electronic wrist band, electronic head band, portable music player, electronic activity tracking device, pedometer, smart-watch, GPS system, and so on. External devices 150 may be connected to the in-vehicle computing system via a communication link 136 which may be wired or wireless, as discussed with reference to communication link 130, and configured to provide two-way communication between the external devices and the in-vehicle computing system. For example, external devices 150 may include one or more sensors andcommunication link 136 may transmit sensor output from external devices 150 to in-vehicle computing system or infotainment system 109 and touch screen 108. External devices 150 may also store and / or receive information regarding contextual data, user behavior / preferences, operating rules, and so on and may transmit such information from external devices 150 to in- vehicle computing system or infotainment system 109 and touch screen 108.

[0030] In-vehicle computing system or infotainment system 109 may analyze the input received from external devices 150. mobile device 128, and / or other input sources and select settings for various in-vehicle systems (such as climate control system or audio system), provide output via touch screen 108 and / or speakers 112, communicate with mobile device 128 and / or external devices 150. and / or perform other actions based on the assessment. In some embodiments, all or a portion of the assessment may be performed by mobile device 128 and / or external devices 150.

[0031] In some embodiments, one or more of external devices 150 may be communicatively coupled to in-vehicle computing system or infotainment system 109 indirectly, via mobile device 128 and / or another of external devices 150. For example, communication link 136 may communicatively couple external devices 150 to mobile device 128 such that output from external devices 150 is relayed to mobile device 128. Data received from external devices 150 may then be aggregated at mobile device 128 with data collected by mobile device 128, the aggregated data then transmitted to in-vehicle computing system or infotainment system 109 and touch screen 108 via communication link 130. Similar data aggregation may occur at a server system and then transmitted to in-vehicle computing system or infotainment system 109 and touch screen 108 via communication link 136 and / or communication link 130.

[0032] FIG. 2 shows a block diagram of an in-vehicle computing system or infotainment system 109 configured and / or integrated inside vehicle 102. In-vehicle computing system or infotainment system 109 may perform one or more of the methods described herein in some embodiments. In some examples, in-vehicle computing system or infotainment system 109 may be a vehicle infotainment system configured to provide information-based media content (audio and / or visual media content, including entertainment content, navigational services, and so on) to a vehicle user to enhance the operator’s in-vehicle experience. In-vehicle computing system or infotainment system 109 may include, or be coupled to, various vehicle systems, sub-systems, hardware components, as well as software applications and systems that are integrated in. or integratable into, vehicle 102 in order to enhance an in-vehicle experience for a driver and / or a passenger. As disclosed herein, the infotainment system 109 may beconfigured to automatically execute a plurality of user assistance operations that are determined based on monitoring user facial movement, including, but not limited to, activating a vehicle function, ceasing a vehicle function, adjusting a vehicle operating condition, and requesting emergency assistance according to the exemplary methods described herein.

[0033] In-vehicle computing system or infotainment system 109 may include one or more processors including an operating system processor 214 and an interface processor 220. Operating system processor 214 may execute an operating system on in-vehicle computing system or infotainment system 109, and control input / output, display, playback, and other operations of in-vehicle computing system or infotainment system 109. Interface processor 220 may interface with a vehicle control system 230 via an inter-vehicle system communication module 222.

[0034] Inter-vehicle system communication module 222 may output data to one or more other vehicle systems 231 and / or one or more other vehicle control elements 261, while also receiving data input from other vehicle systems 231 and other vehicle control elements 261, e.g., by way of vehicle control system 230. When outputting data, inter-vehicle system communication module 222 may provide a signal via a bus corresponding to any status of the vehicle, the vehicle surroundings, or the output of any other information source connected to the vehicle. Vehicle data outputs may include, for example, analog signals (such as current velocity ), digital signals provided by individual information sources (such as clocks, thermometers, location sensors such as Global Positioning System (GPS) sensors, and so on), digital signals propagated through vehicle data networks (such as an engine controller area network (CAN) bus through which engine related information may be communicated, a climate control CAN bus through which climate control related information may be communicated, and a multimedia data network through which multimedia data is communicated between multimedia components in the vehicle). For example, in-vehicle computing system or infotainment system 109 may retrieve from the engine CAN bus the current speed of the vehicle estimated by the wheel sensors, a power state of the vehicle via a battery' and / or power distribution system of the vehicle, an ignition state of the vehicle, and so on. In addition, other interfacing means such as Ethernet may be used as well without departing from the scope of this disclosure.

[0035] A storage device 208 may be included in in-vehicle computing system or infotainment system 109 to store data such as instructions executable by operating system processor 214 and / or interface processor 220 in non-volatile form. Storage device 208 may store application data, including prerecorded sounds, to enable in-vehicle computing system orinfotainment system 109 to run an application for connecting to a cloud-based server and / or collecting information for transmission to the cloud-based server. The application may retrieve information gathered by vehicle systems / sensors, input devices (e.g., a user interface 218), data stored in one or more storage devices, such as a volatile memory 219A or a non-volatile memory 219B, devices in communication with the in-vehicle computing system (e.g., a mobile device connected via a Bluetooth® link), and so on. (Bluetooth® is a registered trademark of Bluetooth SIG. Inc., Kirkland. WA.) In-vehicle computing system or infotainment system 109 may further include a volatile memory 219A. Volatile memory 219A may be random access memory (RAM). Non-transitory storage devices, such as non-volatile storage device 208 and / or non-volatile memory 219B, may store instructions and / or code that, when executed by a processor (e.g.. operating system processor 214 and / or interface processor 220), controls in- vehicle computing system or infotainment system 109 to perform one or more of the actions described in the disclosure.

[0036] A microphone 202 may be included in in-vehicle computing system or infotainment system 109 to receive voice commands from a user, to measure ambient noise in the vehicle, to determine whether audio from speakers of the vehicle is tuned in accordance with an acoustic environment of the vehicle, and so on. A speech processing unit 204 may process voice commands, such as the voice commands received from microphone 202. In some embodiments, in-vehicle computing system or infotainment system 109 may also be able to receive voice commands and sample ambient vehicle noise using a microphone included in an audio system 232 of the vehicle.

[0037] One or more additional sensors may be included in a sensor subsystem 210 of in- vehicle computing system or infotainment system 109. For example, sensor subsystem 210 may include a camera, such as a rear view camera for assisting a user in parking the vehicle and / or a cabin camera for identifying a user (e.g., using facial recognition and / or user gestures). Sensor subsystem 210 of in-vehicle computing system or infotainment system 109 may communicate with and receive inputs from various vehicle sensors and may further receive user inputs. For example, the inputs received by sensor subsystem 210 may include transmission gear position, transmission clutch position, gas pedal input, brake input, transmission selector position, vehicle speed, engine speed, mass airflow through the engine, ambient temperature, intake air temperature, and so on, as well as inputs from climate control system sensors (such as heat transfer fluid temperature, antifreeze temperature, fan speed, passenger compartment temperature, desired passenger compartment temperature, ambient humidity, and so on), an audio sensor detecting voice commands issued by a user, a fob sensorreceiving commands from and optionally tracking the geographic location / proximity of a fob of the vehicle, and so on.

[0038] While certain vehicle system sensors may communicate with sensor subsystem 210 alone, other sensors may communicate with both sensor subsystem 210 and vehicle control system 230, or may communicate with sensor subsystem 210 indirectly via vehicle control system 230. A navigation subsystem 211 of in-vehicle computing system or infotainment system 109 may generate and / or receive navigation information such as location information (e.g., via a GPS sensor and / or other sensors from sensor subsystem 210), route guidance, traffic information, point-of-interest (POI) identification, and / or provide other navigational services for the driver.

[0039] An external device interface 212 of in-vehicle computing system or infotainment system 109 may be coupleable to and / or communicate with one or more external devices 150 located external to vehicle 102. While the external devices are illustrated as being located external to vehicle 102, it is to be understood that they may be temporarily housed in vehicle 102, such as when the user is operating the external devices while operating vehicle 102. In other words, external devices 150 are not integral to vehicle 102. External devices 150 may include a mobile device 128 (e.g., connected via a Bluetooth®, NFC, WI-FI Direct®, or other wireless connection) or an alternate Bluetooth®-enabled device 252. (Wi-Fi Direct® is a registered trademark of Wi-Fi Alliance, Austin, Texas.)

[0040] Mobile device 128 may be a mobile phone, smart phone, wearable devices / sensors that may communicate with the in-vehicle computing system via wired and / or wireless communication, or other portable electronic device(s). Other external devices include one or more external services 246. For example, the external devices may include extra-vehicular devices that are separate from and located externally to the vehicle. Still other external devices include one or more external storage devices 254, such as solid-state drives, pen drives. Universal Serial Bus (USB) drives, and so on. External devices 150 may communicate with in- vehicle computing system or infotainment system 109 either wirelessly or via connectors without departing from the scope of this disclosure. For example, external devices 150 may communicate with in-vehicle computing system or infotainment system 109 through external device interface 212 over a network 260, a USB connection, a direct wired connection, a direct wireless connection, and / or other communication link.

[0041] External device interface 212 may provide a communication interface to enable the in-vehicle computing system to communicate with mobile devices associated with contacts of the driver. For example, external device interface 212 may enable phone calls to be establishedand / or text messages (e.g., Short Message Service (SMS), Multimedia Message Service (MMS), and so on) to be sent (e.g., via a cellular communications network) to a mobile device associated with a contact of the driver. External device interface 212 may additionally or alternatively provide a wireless communication interface to enable the in-vehicle computing system to synchronize data with one or more devices in the vehicle (e.g., the driver’s mobile device) via Wi-Fi Direct®, as described in more detail below.

[0042] One or more applications 244 may be operable on mobile device 128. As an example, a mobile device application 244 may be operated to aggregate user data regarding interactions of the user with the mobile device. For example, mobile device application 244 may aggregate data regarding music playlists listened to by the user on the mobile device, telephone call logs (including a frequency and duration of telephone calls accepted by the user), positional information including locations frequented by the user and an amount of time spent at each location, and so on. The collected data may be transferred by application 244 to External device interface 212 over network 260. In addition, specific user data requests may be received at mobile device 128 from in-vehicle computing system or infotainment system 109 via external device interface 212. The specific data requests may include requests for determining where the user is geographically located, an ambient noise level and / or music genre at the user’s location, an ambient weather condition (temperature, humidity7, and so on) at the user’s location, and so on. Mobile device application 244 may send control instructions to components (e.g., microphone, amplifier, and so on) or other applications (e.g.. navigational applications) of mobile device 128 to enable the requested data to be collected on the mobile device or requested adjustment made to the components. Mobile device application 244 may then relay the collected information back to in-vehicle computing system or infotainment system 109.

[0043] Likewise, one or more applications 248 may be operable on external services 246. As an example, external services applications 248 may be operated to aggregate and / or analyze data from multiple data sources. For example, external services applications 248 may aggregate data from one or more social media accounts of the user, data from the in-vehicle computing system (e.g., sensor data, log files, user input, and so on), data from an internet query (e.g., weather data, POI data), and so on. The collected data may be transmitted to another device and / or analyzed by the application to determine a context of the driver, vehicle, and environment and perform an action based on the context (e.g., requesting / sending data to other devices).

[0044] Vehicle control system 230 may include controls for controlling aspects of various vehicle systems 231 involved in different in-vehicle functions. These may include, for example,controlling aspects of an advanced driver assistance system (ADAS) 238 for monitoring the vehicle cabin and providing assistance to the driver or other vehicle occupants, aspects of vehicle audio system 232 for providing audio entertainment to the vehicle occupants, aspects of a climate control system 234 for meeting the cabin cooling or heating needs of the vehicle occupants, as well as aspects of a telecommunication system 236 for enabling vehicle occupants to establish telecommunication linkage with others.

[0045] Audio system 232 may include one or more acoustic reproduction devices including electromagnetic transducers such as one or more speakers 235. Vehicle audio system 232 may be passive or active such as by including a power amplifier. In some examples, in- vehicle computing system or infotainment system 109 may be a sole audio source for the acoustic reproduction device or there may be other audio sources that are connected to the audio reproduction system (e.g., external devices such as a mobile phone). The connection of any such external devices to the audio reproduction device may be analog, digital, or any combination of analog and digital technologies.

[0046] Climate control system 234 may be configured to provide a comfortable environment within the cabin or passenger compartment of vehicle 102. Climate control system 234 includes components enabling controlled ventilation such as air vents, a heater, an air conditioner, an integrated heater and air-conditioner system, and so on. Other components linked to the heating and air-conditioning setup may include a windshield defrosting and defogging system capable of clearing the windshield and a ventilation-air filter for cleaning outside air that enters the passenger compartment through a fresh-air inlet.

[0047] ADAS 238 may include one or more sensors configured to monitor the vehicle cabin and one or more vehicle occupants such as one or more cameras of the sensor subsystem 210. The one or more cameras may be examples of user-facing cameras. Additionally, or alternatively, the cameras may be examples of one or more of a driver monitoring system (DMS) camera and an occupant monitoring system (OMS) camera, examples of which are described in more detail with reference to FIGS. 3-5. The ADAS 238 may be communicably coupled to one or more other systems of the vehicle 102 via the infotainment system 109. In one example, the non-volatile memory 219B of infotainment system 109 may store instructions that when executed cause the operating system processor 214 to execute one or more methods for facial movement monitoring by the ADAS 238. When executed, the instructions may cause the operating system processor 214 to monitor in a first operating mode, via a user-facing camera, eye movement of a user over time and predict, via a psychophysiological state prediction model, one or more psychophysiological states of the user based on the eye movement. The operatingsystem processor 214 may be configured to, in response to an indication of greater than a first threshold psychophysiological state, monitor in a second monitoring mode, via the user-facing camera, at least one of eye movement and lip movement of the user over time, determine, via the code reading model, one or more user requests based on the at least one of eye movement and lip movement, and execute one or more user assistance operations based on the one or more determined user requests. Exemplary embodiments of systems and methods for operating the ADAS 238 to monitor facial movement of vehicle occupants to predict psychophysiological states and determine user requests are described in more detail herein with reference to FIGS. 3-8.

[0048] Vehicle control system 230 may also include controls for adjusting the settings of various vehicle control elements 261 (or vehicle controls, or vehicle system control elements) related to the engine and / or auxiliary elements within a cabin of the vehicle, such as one or more steering wheel controls 262 (e.g., steering wheel-mounted audio system controls, cruise controls, windshield wiper controls, headlight controls, turn signal controls, and so on), instrument panel controls, microphone(s), accelerator / brake / clutch pedals, a gear shift, door / window controls positioned in a driver or passenger door, seat controls, cabin light controls, audio system controls, cabin temperature controls, and so on. Vehicle control elements 261 may also include internal engine and vehicle operation controls (e.g., engine controller module, actuators, valves, and so on) that are configured to receive instructions via the CAN bus of the vehicle to change operation of one or more of the engine, exhaust system, transmission, and / or other vehicle system. The control signals may also control audio output at one or more speakers 235 of vehicle audio system 232. For example, the control signals may adjust audio output characteristics such as volume, equalization, audio image (e g., the configuration of the audio signals to produce audio output that appears to a user to originate from one or more defined locations), audio distribution among a plurality of speakers, and so on. Likewise, the control signals may control vents, air conditioner, and / or heater of climate control system 234. For example, the control signals may increase delivery' of cooled air to a specific section of the cabin.

[0049] Control elements positioned on an outside of a vehicle (e.g., controls for a security system) may also be connected to in-vehicle computing system or infotainment system 109, such as via inter-vehicle system communication module 222. The control elements of vehicle control system 230 may be physically and permanently positioned on and / or in the vehicle for receiving user input. In addition to receiving control instructions from in-vehicle computing system or infotainment system 109, vehicle control system 230 may also receive input fromone or more external devices 150 operated by the user, such as from mobile device 128. This allows aspects of vehicle systems 231 and vehicle control elements 261 to be controlled based on user input received from external devices 150. Further, aspects of vehicle systems 231 and vehicle control elements 261 may be controlled by the ADAS 238 to execute one or more user assistance operations determined based on monitoring user facial movement.

[0050] In-vehicle computing system or infotainment system 109 may further include one or more antennas 206. The in-vehicle computing system may obtain broadband wireless internet access via antennas 206, and may further receive broadcast signals such as radio, television, weather, traffic, and the like. In-vehicle computing system or infotainment system 109 may receive positioning signals such as GPS signals via antennas 206. The in-vehicle computing system may also receive wireless commands via radio frequency (RF) such as via antennas 206 or via infrared or other means through appropriate receiving devices. In some embodiments, antenna 206 may be included as part of audio system 232 or telecommunication system 236. Additionally, antenna 206 may provide AM / FM radio signals to external devices 150 (such as to mobile device 128) via external device interface 212.

[0051] One or more elements of in-vehicle computing system or infotainment system 109 may be controlled by a user via user interface 218. User interface 218 may include a graphical user interface presented on a touch screen, such as touch screen 108 and / or display screen 111 of FIG. 1, and / or user-actuated buttons, switches, knobs, dials, sliders, and so on. For example, user-actuated elements may include steering wheel controls, door and / or window controls, instrument panel controls, audio system settings, climate control system settings, and the like. A user may also interact with one or more applications of in-vehicle computing system or infotainment system 109 and mobile device 128 via user interface 218. In addition to receiving a user's vehicle setting preferences on user interface 218, vehicle settings selected by in-vehicle control system 230 may be displayed to a user on user interface 218. Notifications and other messages (e.g., received messages), as well as navigational assistance, may be displayed to the user on a display of the user interface. User preferences / information and / or responses to presented messages may be performed via user input to the user interface.

[0052] Referring to FIG. 3, a block diagram of a psychophysiological state and user request detection process 300 is depicted, illustrating an embodiment of a system for predicting psychophysiological states and determining user requests based on facial movement metrics of a user. Operation of the process provides predictions of emotional and cognitive states of an individual such levels of arousal, drowsiness, stress, etc., and determinations of a plurality eye blinking or lip reading messages corresponding to user requests.

[0053] The video capture device 304 is configured to capture video footage of the face of user 302. In one or more examples, the user 302 may be positioned in a vehicle, and the video capture device 304 may be a user-facing camera of the vehicle, and such as the vehicle 102 in FIGS. 1-2. In some examples, the video capture device 304 may be a near-infrared (NIR) camera with a framerate of 60 frames-per-second (fps), a view angle of 50-60 degrees, and a resolution of 3280x752. The camera is positioned to have a clear view of user 302's face, particularly the eye region, to facilitate accurate data capture. In another example, additionally, or alternatively, the camera is positioned to have a clear view of user 302's mouth region, to facilitate accurate data capture of the mouth region. In another embodiment, the video capture device 304 may include advanced sensors capable of micromovement registration, enhancing the precision of the captured data. The video footage captured by the video capture device 304 serves as the raw data from which facial movement characteristics are extracted. In some examples, the framerate of video capture device 304 may be adjusted based on a monitoring mode. As one non-limiting example, the framerate may be 60 fps when operating the camera to predict psychophysiological states, and may be may be higher than 60 fps when operating the camerato determine user requests.

[0054] Facial movement determination 306 segments the position of facial features of interest in each frame of the video footage and determines facial movement data for each frame. In one or more examples, facial movement determination 306 may involve analysis of one or both of eye movement such as movement of the eyes and eyelids, and lip movement such as movement of the lips, mouth, teeth, tongue, and other parts of the face that are involved in speech production. In particular, facial movement determination 306 may include eye gaze vector and eyelid position determination, which segments eyelid positions in each frame of the video footage and determines eye gaze vectors for each frame. In one embodiment, this determination may involve the use of a eyetracker module or similar toolkit that derives eye gaze information from raw NIR video frames. The module may perform operations such as facebox detection (e.g., detecting the region of the video including the face), eyebox detection (e.g., detecting the region of the video including the eyes), head position detection, eyelid position detection, pupil detection (size and position), and the determination of the eye gaze vector itself.

[0055] In additional or alternative examples, facial movement determination 306 may include mouth and / or lip shape and lip position determination, which segments lip positions in each frame of the video footage and determines mouth and / or lip shapes for each frame. This determination may involve the use of a liptracker module or similar toolkit that derives mouthand / or lip shape information from raw NIR video frames. The module may perform operations such as facebox detection, mouthbox detection, head position detection, lip position detection, tongue detection (position), teeth detection (position), and the determination of the mouth and / or lip shape itself. As used herein, lip movement, lip movement behaviors, lip movement data, and related terms may be understood to generally refer to the movement of the parts of the face that are involved in speech production, such as, but not limited to. lips, mouth, teeth, and tongue.

[0056] In further examples, the facial movement determination 306 may employ machine learning algorithms to enhance the accuracy of the segmentation and vector determination. For example, machine learning algorithms may train on user-specific facial features, challenging lighting conditions, or variable head movement conditions.

[0057] Second order metric determination 308 determines a plurality of second order facial movement metrics from the facial movement data obtained from the facial movement determination 306. In one or more examples, second order metric determination 308 may determine a plurality of second order eye movement metrics from the eye gaze vectors and eyehd positions obtained from eye gaze vector and eyelid position determination. These metrics are indicative of discrete eye behaviors such as saccades, fixations, blinks, and long closures. The second order metric determination 308 may calculate metainformation associated with each discrete eye behavior, including the length of the behavior, average gaze movement speed, and gaze fluctuation boundaries. In another example, the second order metric determination 308 may identify blinks based on eyelid openness levels, where a blink comprises a contiguous portion of frames from the sequence of eyelid openness levels satisfying a blink criterion based on calculated differences between adjacent eyehd openness levels for each frame. Further, second order eye movement metrics may include a sequence of eye movements, such as a number of blinks, duration of each blink, and / or a pattern of blinks.

[0058] In one or more examples, second order metric determination 308 may determine a plurality of second order lip movement metrics from the lip shape and lip position determination. These metrics are indicative of lip movement behaviors that are involved in speech production such as a shape formed by the lips, a distance between lips, lips touching, tongue contact with lips, tongue contact teeth, and so on. The second order metric determination 308 may calculate metainformation associated with each lip behavior, including the length of the behavior. The second order metric determination 308 may identify' discrete speech units based on second order lip movement metrics, where a discrete speech unit comprises a contiguous portion of frames from the sequence of lip positions, or other speechproduction behavior, satisfying a discrete speech unit criterion based on calculated differences between adjacent lip positions for each frame. In one example, discrete speech units may be understood to represent single syllables. Further, second order lip movement metrics may include a sequence of lip movements, such as a number or pattern of lip movements indicating compound speech units, e.g., multi-syllable words or phrases.

[0059] Machine readable metric determination 309 converts the sequence of second order facial movement metrics into a machine readable representation of facial movement behaviors using a sliding time window. In one or more examples, the machine readable metric determination 309 may convert the sequence of discrete eye behaviors into a machine readable representation of eye behaviors using a sliding time window, e.g., a first sliding time window. Similarly, in some examples, the machine readable metric determination 309 may convert the sequence of lip movements into a machine readable representation of lip behaviors using a sliding time window, e.g., a second sliding time window. The sliding time window, e.g., the first sliding time window, the second sliding time window, etc., may be determined via a calibration process. The transformation prepares the data for analysis by one or more facial movement interpretation models 330. In one example, machine readable metric determination 309 may include calculating a number of blinks, duration of each blink, a number of fixations, and an overall fixation time within the pre-determined time window. In one example, machine readable metric determination 309 may include a representation of the syllable associated with a discrete speech unit and calculating a number of discrete speech units within the predetermined time window'. In another example, machine readable metric determination 309 may involve the use of continuous metrics such as the number of saccades, the overall saccade time, and the average saccade time, providing anuanced representation of user 302's eye movement patterns over time. Similarly, continuous metric determination may be applied to provide machine readable representation of compound speech units.

[0060] Facial movement interpretation models 330 comprises a suite of machine learning models, including psychophysiological state prediction models 332 and code reading model 322, which are configured to respectively predict psychophysiological states and determine user requests using the machine readable representations of the plurality of second order facial movement metrics. In one or more examples, psychophysiological state prediction models 332 may include a first model 314 through to an Nth model 316, each configured to predict one or more psychophysiological states using the machine readable representation of the plurality of second order eye movement metrics. In one example, first model 314 may be a neural networkbased classifier detecting the probability of acute stress from the machine readablerepresentation of the second order eye movement metrics. Additionally, the psychophysiological state prediction models are capable of generating output as per a more generalized state definition, where states of sleepiness, drowsiness, relaxed state, strained state, and stress can be represented as specific levels of arousal. In another example, Nth model 316 may be a deep learning model trained to predict drowsiness of the user 302, which could also be interpreted as a particular level of arousal. In some examples, psychophysiological state prediction models 332 may include a single model; therefore, N may equal I. In such embodiments, the single model may be configured to predict one, or a plurality of psychophysiological states, including an estimated arousal level that encompasses various states of user 302.

[0061] Code reading model 322 may include at least one of an eye movement reading model configured to determine user requests using the machine readable representation of the plurality of second order eye movement metrics and a lip reading model configured to determine user requests using the machine readable representation of the plurality of second order lip movement metrics. In one or more examples, the eye movement reading model may be configured to detect one or more of a plurality of messages in Morse code corresponding to user requests, where paterns of user eye blinks constitute Morse code signals. For example, the machine readable representation of a long duration blink may represent a dash and the machine readable representation of a short duration blink may represent a dot. The machine readable representation of a sequence of long duration blinks and short duration blinks may represent a word or a phrase such as a user request for assistance, e g., "SOS".

[0062] In one or more examples, the lip movement reading model may be configured to detect one or more of a plurality of lip reading messages corresponding to user requests, where paterns of user lip movements may represent speech sounds or syllables. As one non-limiting example, the machine readable representation of a first mouth shape held for a first duration may represent an “S”, a second mouth shape held for a second duration may represent an “O”, and so on. The machine readable representation of a sequence of mouth shapes durations may represent a word or a phrase such as a user request for assistance, e g., ‘‘SOS'’.

[0063] In some examples, the code reading model 322 may be trained on machine readable representations of user-specific facial movement data corresponding to a plurality of user requests. For example, the code reading model 322 may be trained on user-specific video data corresponding to the user demonstrating one or both of eye movement and lip movement behaviors. For example, a user may demonstrate a blink, long blink, a short blink, a sequence of blinks, and so on. Similarly, a user may demonstrate forming speech sounds correspondingspecific user requests such as “SOS"’, “help me'’, “open trunk”, “open window”, and so on. In additional or alternative examples, the code reading model 322 may be trained on machine readable representations of situational data corresponding to events that may arise during vehicle operation such as potentially dangerous or threatening situations, e.g., emergency situations. For example, the code reading model 322 may be trained on video footage corresponding to scenarios such as the presence of a potentially dangerous object in the vehicle cabin, threatening behavior of a vehicle occupant, or other scenarios, where each scenario corresponds to one or more machine readable representations of situations, e.g., a first situation, a second situation, and so on.

[0064] The output of the psychophysiological state prediction models 332 includes predictions from a first prediction 318 through an Nth prediction 320, which represent the predicted psychophysiological states of user 302. In one example, first prediction 318 may indicate a binary state, such as the presence or absence of acute stress, while Nth prediction 320 may provide a probabilistic assessment of user 302's drowsiness levels. Furthermore, in some examples, the output of the psychophysiological state prediction models 332 model may comprise an estimated arousal level, which may be used directly as a measure of human state or further processed, for example, by applying specific thresholds, to determine if a subject is in a particular state such as drowsiness or stress. The estimated arousal level may be utilized either as a direct measure or to infer specific states through additional logic. In another example, the predictions may be used to trigger alerts or interventions in real-time, such as in a driver monitoring system (DMS) or operator monitoring system (OMS), where the arousal level itself can be a parameter for ADAS intervention. In one example, the output may trigger a shift to operation of the process in a first mode or a second mode, the shift based on additional logic, which is described in more detail below, such as with reference to FIGS. 5-6. As one nonlimiting example, the output of the Nth prediction 320 indicating greater than threshold stress may shift to the process to using the code reading mode to determine user requests.

[0065] The output of the code reading model 322 includes a code determination 324, which represents the determined user requests of user 302. In one or more examples, code determination 324 may indicate a user request to execute one or more user assistance operations such as activating a vehicle function, ceasing a vehicle function, adjusting a vehicle operating condition, and requesting emergency assistance. In one or more examples, such as in response to the code reading model 322 detecting a first situation, the output may include automatically generating a user notification, and in response to detecting a second situation, the output mayinclude automatically generating an emergency assistance request comprising a vehicle license number, location, and image.

[0066] The psychophysiological state and user request detection process 300, as described herein, enables substantially real-time detection of psychophysiological states and user requests by analyzing facial movement data. The disclosed techniques allow for the transformation of facial movement metrics into a compact, quantitative, and machine readable representation that can be effectively utilized by machine learning models, including models configured to detect user requests in eye blinking-signaled Morse code and lip reading messages, thereby enabling the user to command user assistance operations and convey messages discreetly, and increasing overall user control of the vehicle control system.

[0067] Referring to FIG. 4. a system 400 for predicting user psychophysiological states and determining user requests based on facial metrics is shown. The system 400 is configured to analyze facial movement data corresponding to user requests, which may be referred to as a request determination mode, and is configured to analyze facial movement data and predict various psychophysiological states, such as cognitive load or stress, which may be referred to as a state prediction mode. In one or more examples, the system 400 may operate in conjunction with a logic that shifts the system 400 from operating in the request determination mode to operating in the state prediction mode. One example of the logic is described in more detail with reference to FIG. 6.

[0068] The system 400 includes a facial movement interpretation device, hereinafter a device 402. The device 402 orchestrates the acquisition, processing, and analysis of facial movement data to predict a psychophysiological state or determine a user request of a vehicle occupant, such as the driver. In one example, the device 402 may be integrated into an ADAS of a vehicle, such the vehicle 102 in FIGS. 1-2. The device 402 may be used to monitor one or both of the eye movement and lip movement of the driver. For example, under some conditions, the device 402 may monitor the eye movement of the driver to predict the driver's psychophysiological state, and under different conditions, the device 402 may monitor the eye movement and / or lip movement to detect one or more of a plurality of messages in Morse code or lip reading.

[0069] The processor 404 within the device 402 may be a multi-core processor capable of parallel processing to ensure real-time data analysis. In one embodiment, the processor 404 may be a specialized processor optimized for machine learning tasks, enabling the efficient execution of facial movement interpretation models. In another embodiment, the processor 404may be a general-purpose processor that is part of a larger distributed computing system, such as a cloud-based service, where facial movement data is processed remotely.

[0070] Non-transitory memory 406 is a component of the device 402 that stores machine- readable instructions for the operation of the system 400. The non-transitory memory 406 houses several modules that may be employed in the interpretation of facial movement. The non-transitory memory 406 may store a first order facial movement metrics module 408, which is responsible for determining first order facial movement metrics for each frame of video footage acquired by the video capture device 440. In one example, the first order facial movement metrics module 408 may be configured to determine eye gaze vectors and eyelid openness levels for each frame of acquired image data. In additional or alternative examples, the first order facial movement metrics module 408 may be configured to determine lip shapes and lip positions for each frame of acquired image data. The non-transitory memory 406 may store a second order facial metrics module 410 that derives second order facial movement metrics from the first order facial movement metrics. For example, the second order facial metrics module 410 may be configured to derive second order eye movement metrics from the first order eye movement metrics, the second order eye movement metrics indicative of discrete eye behaviors such as blinks, saccades, and fixations. In another example, the second order facial metrics module 410 may be configured to derive second order lip movement metrics from the first order lip metrics, the second order lip metrics indicative of lip movement behaviors that are involved in speech production.

[0071] The machine readable metric module 412, also stored within the non-transitory memory 406, is configured to transform the second order facial movement metrics into a machine readable representation within a predetermined time window. In one example, the machine readable metric module 412 may enable the system to calculate numerical metrics such as the number of blinks, the duration of each blink, number of fixations, and overall fixation time, which may be used for training machine learning algorithms to predict psychophysiological states. Additionally, or alternatively, the machine readable metric module 412 may enable the system to represent long blinks, short blinks, and sequences of blinks, which may be used for training machine learning algorithms to detect Morse code messages corresponding to user requests. Additionally, or alternatively, the machine readable metric module 412 may enable the system to represent lip shapes corresponding to discrete speech units or syllables, and sequences of lip shapes corresponding to compound speech units, words or phrases, which may be used for training machine learning algorithms to detect one or more of a plurality of lip reading messages corresponding to user requests. In one example, themachine readable metric module 412 may utilize sliding time windows to generate metrics associated with every time unit, providing a nuanced view of the user's facial movement over time.

[0072] Facial movement interpretation models 414 are a set of mathematical or machine learning models stored within the non-transitory memory 406. The models are trained to interpret the machine readable representation of facial movement behaviors. In one example, facial movement interpretation models 414 may include one or more state prediction models 416 and one or more of code reading models 420.

[0073] Code reading models 420 are trained to interpret one or both of eye movement data and lip movement data. For example, the code reading models 420 may be trained to map the machine readable representation of eye behaviors to one or more of a plurality of messages in Morse code corresponding to user requests. Additionally, or alternatively, the code reading models 420 may be trained to map the machine readable representation of lip behaviors to one or more of a plurality of lip reading messages corresponding to user requests. In one example, the code reading models 420 may include deep neural networks that leam correlations between user-specific facial movement metrics and corresponding user requests. In additional or alternative examples, the code reading models 420 may leam to detect events that may arise during vehicle operation such as potentially dangerous or threatening situations, e.g., emergency situations.

[0074] The state prediction models 416 are trained to map the machine readable representation of eye behaviors to one or more psychophysiological states. In one example, the state prediction models 416 may include deep neural networks that leam correlations between the metrics and the ground truth of the subject's mental state. In another example, the models may be based on support vector machines or other statistical learning methods that are capable of inferring states such as cognitive load or acute stress from the second order eye movement metrics.

[0075] The display device 430 is an output component of the system 400 that in some examples may present the results of the facial movement interpretation to the user. In one example, the display device 430 may be a dashboard-mounted screen in a vehicle that alerts the driver to changes in their psychophysiological state. In another embodiment, the display device 430 may alert the driver to a detection of a potentially dangerous or threatening situation. In other examples, the output may not be presented via the display device 430.

[0076] The external device 460 is an output component of the system 400 that in some examples may transmit the results of the facial movement interpretation to an external party.In one example, the external device 460 may be connected to the system 400 via a wireless communication link configured to provide two-way communication between the external device 460 and the system 400, such as via the infotainment system 109 in FIGS. 1-2. In one example, the external device 460 may include a communication system of a driver assistance system, the driver assistance system configured to receive driver requests for assistance and transmit the driver requests to an appropriate service, such as police, fire, or medical assistance.

[0077] The video capture device 440 captures video footage of the user from which the first order facial movement metrics are determined. In one embodiment, the video capture device 440 may be a near-infrared (NIR) camera with a high framerate, to ensure accurate and detailed capture of facial movements. The framerate may be adjusted based on the mode of monitoring. In another example, the video capture device 440 may include sensors configured for micromovement registration, which may be used in the prediction of psychophysiological states and the detection of potentially threatening or dangerous situations.

[0078] The user input device 450 allows for user interaction with the system 400. In one example, the user input device 450 may be a touchscreen interface through which training protocols for user-specific facial movement interpretation may be accessed and system settings may be adjusted. For example, the user input device 450 may allow a user to access protocols for training the system on Morse code messages from user-specific eye movement data, or lip reading messages from user-specific lip movement data.

[0079] FIG. 5 shows an example ADAS 500 that may be implemented in a vehicle for driver assistance, such as the vehicle of FIGS. 1-2. In one or more examples, the ADAS 500 operates in coordination with a facial movement interpretation device according to a facial movement interpretation process, such as the process 300 and the device 402 respectively described with reference to FIG. 3 and 4.

[0080] The ADAS 500 includes a camera, such as one or both of a driver monitoring system (DMS) camera and an occupant monitoring system (OMS) camera. The ADAS 500 is trained to monitor the vehicle and the occupants, via the DMS and OMS cameras, to detect one or more conditions of the vehicle operation. Examples include monitoring the facial movement of the driver and monitoring the vehicle cabin. In one or more examples, the ADAS 500 may monitor the vehicle and the occupants via the camera in a first monitoring mode 520 and a second monitoring mode 522. For example, the ADAS may operate in the first monitoring mode 520 under normal operating conditions 504 and may operate in the second monitoring mode 522 in response to an indication of a suspicious situation 502.

[0081] Detection of the suspicious situation 502 and normal operating conditions 504 may involve operation of the facial movement detection device (e.g., the device 402 in FIG. 4). In one or more examples, a suspicious situation 502 may be indicated based on a predicted psychophysiological state of the driver greater than a first threshold, such as greater than threshold predicted level of stress. Normal operating conditions 504 may be indicated based on the predicted psychophysiological state of the driver less than the first threshold, such as less than threshold predicted level of stress. In another example, the suspicious situation 502 may be indicated based on a suspicious object detection in the vehicle cabin such as a sharp object or other potentially dangerous object. In another example, the suspicious situation 502 indication may be based on other sensor signals, such as a signal indicating unusual driver or occupant behavior, or unusual vehicle conditions. For example, the suspicious situation 502 may be indicated in response to a vehicle impact greater than a vehicle impact threshold or a vehicle stop followed by a passenger entry to the vehicle occurring within a duration threshold. Conversely, the normal operating conditions 504 may include the absence of indications of threatening objects, unusual driver behavior, or unusual vehicle conditions.

[0082] In one or more examples, detection of normal operating conditions 504 may cause the ADAS 500 to automatically operate in the first monitoring mode 520. As described above, the first monitoring mode 520 may include monitoring eye movement behaviors to predict emotional and cognitive states of a vehicle occupant, such as the driver. For example, the first monitoring mode 520 may include monitoring the driver to predict emotional and cognitive states indicating levels of arousal, drowsiness, stress, etc., using one or more psychophysiological state prediction models (e.g., psychophysiological state prediction models 332 in FIG. 3, state prediction models 416 in FIG. 4). In some examples, fromnormal operating conditions 504 in the first monitoring mode 520, the driver may command the ADAS 500 to operate in the second monitoring mode 522 such as by selecting a vehicle control setting via the infotainment system (e.g., infotainment system 109 in FIGS. 1-2). As one example, such operation may be useful under conditions where hands-free, discreet command of vehicle control operations are desired by the driver.

[0083] In one or more examples, detection a suspicious situation 502 may cause the ADAS 500 to automatically operate in the second monitoring mode 522. As described above, the second monitoring mode 522 may include monitoring facial movements of the driver to detect user requests using on one or more code reading models (e.g., code reading model 322 in FIG. 3, code reading models 420 in FIG. 4). In one example the second monitoring mode 522 may include operating the ADAS 500 to detect eye movement behaviors indicating messages inMorse code corresponding to user requests. In other examples, the second monitoring mode may include operating the ADAS 500 to detect lip movement behaviors indicating lip reading commands corresponding to user requests. The ADAS 500 may detect lip reading in various languages such as English and German.

[0084] With ADAS 500 operating in the second monitoring mode 522, one or both of the driver and other vehicle occupants may command one or more user assistance operations using eye blinking or lip reading commands, as described above and with reference to FIGS. 3-4. For example, Morse code commands may be defined based on sequences long blinks and short blinks. Lip reading commands may be defined based on sequences of mouth shapes corresponding to spoken syllables, words, or phrases. As a few examples, user assistance operations and corresponding commands may be defined for opening and closing vehicle doors, windows, trunk, or hood 508, commanding on or off of the windshield wipers 510, climate control operation 512, on or off of high beams, picking up, ending or recording a call, playing music, or other functions.

[0085] In some examples, the ADAS 500 may be configured to automatically transmit a request for emergency driver assistance 506 (e.g., an ‘‘SOS’7) in response to detection of an emergency situation, which may include one or more of the indications of a suspicious situation 502, such as the detection of an unusual vehicle condition or a threatening situation. In other examples, additionally, or alternatively, the ADAS 500 may be configured to transmit the request for emergency driver assistance 506 based on a determined user request, such as a lip reading or eye blinking command corresponding to "SOS". In one example, the request for emergency driver assistance 506 may include transmitting a call for help to an emergency service 514, such as police, ambulance, or fire service patrolling in the area, including vehicle location and license plate number. The call for help may be a silent call.

[0086] A flowchart of a method 600 for an ADAS for operating in a first monitoring mode in response to a first set of conditions and operating in a second monitoring mode in response to a second set of conditions is shown in FIG. 6. The method 600 may be employed by a facial movement interpretation system of the ADAS to shift between the first monitoring mode, where eye movement behaviors are assessed to predict the mental and emotional state of a user, and the second monitoring mode, where eye movement behaviors or lip movement behaviors are assessed to determine user requests. In one or more examples, the method 600 may be employed by the ADAS 500 described with reference to FIG. 5. Instructions for carrying out the method 600 and the other methods described herein may be executed by an in-vehicle computing system based on computer readable instructions stored on a memory of the in-vehicle computing system and in conjunction with signals received from sensors of the vehicle system, such as the OS processor 214, the non-volatile memory 219B, and the sensor subsystem 210 included in the infotainment system 109 in FIG. 1 and FIG. 2, and the processor 404 and non-transitory memon 406 included in the system 400 in FIG. 400. The in-vehicle computing system may employ actuators of the vehicle system, such as the plurality' of controls of the vehicle control system 230. to adjust vehicle system operation, according to the methods described below.

[0087] At 602, the method 600 may include receiving and / or determining vehicle operating conditions. Vehicle operating conditions may include presence and location of vehicle occupants, such as indicated by cameras, seat-mounted pressure sensors, or microphones, door activity indicated by door sensors, vehicle settings, such as cruise control, or autonomous driving, vehicle speed, vehicle orientation, direction of travel, location indicated by a GPS, and so on.

[0088] At 604, the method 600 may include monitoring a user-facing camera in a first monitoring mode. In one example, the ADAS may monitor the vehicle and its occupants in the first monitoring mode under a first set of conditions, such as normal operating conditions. For example, the first monitoring mode may be considered a default mode. The first monitoring mode may include monitoring the user-facing camera to predict psychophysiological states of a vehicle occupant, such as the driver, based on eye movement metrics. For example, the first monitoring mode may include use of a facial movement interpretation system to assess the mental and emotional state of the driver by analyzing eye movement data indicative of various psychophysiological states. A flow chart illustrating an example of a method for monitoring the user-facing camera in the first operating mode is described below with reference to FIG. 7.

[0089] At 606, the method 600 may include determining whether greater than a first threshold psychophysiological state of the vehicle occupant is indicated. In one or more examples, the psychophysiological state may include a predicted level of stress. The first threshold may be a non-zero, positive value threshold. The first threshold may be determined from a calibration process. In one example, the first threshold may include an amount of a machine representation of eye movement behaviors occurring within a pre-determined time window which indicate a level of stress. For example, the first threshold may include of number of blinks within a pre-determined time window. In another example, the first threshold may include a number of saccades within a pre-determined time window.

[0090] In response to determining greater than the first threshold psychophysiological state of the vehicle occupant is indicated, at 608, the method 600 may include monitoring the user-facing camera in a second monitoring mode. In one example, the ADAS may monitor the vehicle and its occupants in the second monitoring mode under a second set of conditions. One example of the second set of conditions may include the indication of greater than the first threshold psychophysiological state. However, other conditions are considered. In another example, the second set of conditions may include a suspicious situation detection, which may include one or more of the indicators described above with reference to FIG. 5, including, but not limited to, a vehicle impact greater than a vehicle impact threshold, a vehicle stop followed by a passenger entry to the vehicle occurring within a duration threshold, a suspicious object detection, or other situation on which the ADAS may be trained to detect. The second monitoring mode may include monitoring the user-facing camera to determine user requests based on facial movement metrics. For example, the second monitoring mode may include use of the facial movement interpretation system to determine user requests by analyzing facial movement data indicative of user commands. A flow chart illustrating an example of a method for monitoring the user-facing camera in the second operating mode is described below with reference to FIG. 8.

[0091] In response to determining greater than the first threshold psychophysiological state of the user is not indicated, the method 600 may include determining whether less than a second psychophysiological state of the user is indicated at 610. In one or more examples, the second psychophysiological state may include a predicted level of alertness, where less than the second threshold indicates drowsiness. The second threshold may be a non-zero, positive value threshold. The second threshold may be determined from a calibration process. Similarly, the second threshold may include an amount of a machine representation of eye movement behaviors occurring within a pre-determined time window which indicate a level of drowsiness, such as a number of blinks. In one example, the first threshold may include of number of blinks within a pre-determined time window. In another example, the first threshold may include a number of closures w ithin a pre-determined time window.

[0092] In response to determining less than the second threshold psychophysiological state of the vehicle occupant is indicated, at 612, the method 600 may include generating a user notification and displaying the user notification via the infotainment system of the vehicle. For example, the user notification may be displayed via a dashboard-mounted screen in the vehicle that alerts the driver to changes in their psychophysiological state. In another example, theuser notification may include an audio signal, such as a loud pulse, or a tactile signal, such as a vibration.

[0093] In response to determining less than the second threshold psychophysiological state of the user is not indicated, the method 600 may return to 604 for monitoring the user-facing camera in the first monitoring mode. In one or more examples, the method 600 may continuously monitor the user-facing camera in the first monitoring mode so long as a psychophysiological state greater than the first threshold is not detected.

[0094] Referring to FIG. 7, a flowchart of a method 700 for predicting psychophysiological states of a user based on eye movement metrics is shown. The method 700 may be employed by a facial movement interpretation system to assess the mental and emotional state of a vehicle occupant, such as a driver, by analyzing eye movement data indicative of various psychophysiological states. The method 700 may be executed by an ADAS that is configured to operate in a first operating mode or a second operating mode, and may be implemented as part of the logic for selecting the operating mode of the ADAS.

[0095] At 702, the method 700 may include capturing video footage of a user via a userfacing camera at a first framerate. The camera, which may be a Near-Infrared (NIR) camera with a first framerate of 60 FPS, captures a sequence of images over time, providing raw data for subsequent analysis. In one example, the camera may be integrated into a DMS or an OMS to monitor the eye movements of one or more vehicle occupants in real-time. In one example, the first framerate may be calibrated for analyzing eye movement data to detect one or more psychophysiological states.

[0096] At 704, the method 700 may include determining eye gaze vectors and eyelid openness levels for each frame of the video footage. This involves processing the captured images to extract the direction of the "virtual" vector coming from the center of the pupil and being normal to the surface of the eye, as well as the maximum distance between the upper and lower eyelids. In one embodiment, the eye gaze vectors may be represented as Euler angles or coordinates in three-dimensional space. In another embodiment, the eyelid openness levels may be measured in metric or angular coordinate systems, or presented as a degree of openness relative to the maximum possible level for each individual.

[0097] At 706, the method 700 may include determining a plurality of second order eye movement metrics from the eye gaze vectors and eyelid openness levels over a pre-determined time window. The second order eye movement metrics define eye behavioral patterns in time and may include fixations, saccades, blinks, and long closures. Fixations are periods where the eye gaze remains relatively stable, indicating focused attention, and can be defined as the eyegaze being maintained within a small angular area, such as a 2-degree angle segment, for a duration exceeding a threshold, such as 100 milliseconds. Saccades are rapid eye movements that occur as the eye jumps from one point of interest to another, and their patterns may be indicative of underlying psychophysiological states. In one example, saccades are identified by detecting eye movements that exceed a certain angular velocity7threshold. In another example, the method may also detect microsaccades, which are smaller, involuntary saccades that occur during fixations. Long closures, which may indicate drowsiness, are identified as periods where the eyelids remain closed for an extended duration, such as more than one second. In some embodiments, at operation 706 the system calculates metainformation for each behavioral pattern, including the length of the event, average gaze movement speed, and gaze fluctuation boundaries.

[0098] At 708, the method 700 may include transforming the plurality of second order eye movement metrics into a machine readable representation for the pre-determined time window. The transformation includes converting the discrete sequence of saccades, fixations, and blinks into a numerical metric that defines the "amount" of patterns that occurred during the time window. In one example, the machine readable representation may include characteristics such as the number of blinks, number of fixations, overall fixation time, and average fixation time. In another example, the window may be sliding, allowing for the evaluation of numerical metrics within overlapping time windows, thereby providing metrics associated with every time unit.

[0099] At 710, the method 700 may include predicting one or more psychophysiological states for the user based on the machine readable representation of the plurality of second order eye movement metrics. The prediction is facilitated by a machine learning model, which may include deep neural networks that leam correlations between the metrics and the ground truth of the user's psychophysiological state. In one example, the model may be configured to infer whether a user is experiencing cognitive load or acute stress. In another example, the model may be trained using machine readable representations derived from eye movement data to accurately predict states such as drowsiness or distraction.

[0100] At 712, the method 700 may include outputting the one or more predicted psychophysiological states. In one example, outputting may include displaying the predicted states on a user interface or transmitting the information to a vehicle's central processing unit for real-time intervention. In one example, the output may be used to alert the user or activate safety7measures in response to detected states such as fatigue or high cognitive load. In one ormore examples, the output may be used to shift the facial movement monitoring system to the second monitoring mode. Following 712, method 700 may end.

[0101] Referring to FIG. 8, a flowchart of a method 800 for determining one or more user requests based on facial movement metrics is shown. The method 800 may be employed by a facial movement interpretation system to determine user requests of a vehicle occupant, such as a driver, by analyzing facial movement data corresponding to a plurality of user requests. In one or more examples, the method 800 may be executed by an ADAS that is configured to operate in a first operating mode or a second operating mode, and may be implemented in response to an indication of greater than a first threshold psychophysiological state of a vehicle occupant, such as the driver, as described above. Additionally, or alternatively, the method 800 may be implemented in response to an indication of a second set of conditions, such as in response to a suspicious situation detection. The method 800 may be implemented as part of the ADAS as a hands-free and discreet approach for activating user assistance.

[0102] At 802, the method 800 may include determining whether an emergency situation is indicated, e.g.. an “SOS”. In one example, the emergency situation may be determined in response to detection of a distress signal greater than a threshold distress. Distress may be indicated variously, and the ADAS may be trained to detect distress using one or more or a plurality of sensors and models. In one or more examples, the emergency situation may be determined in response to a suspicious situation detection as described above. For example, unusual vehicle behavior may indicate the emergency situation as a sensor signal indicating the vehicle is not upright, a vehicle impact greater than an impact threshold, or unusual vehicle occupant behavior, such as a microphone signal indicating speech volume above a threshold, and so on. In one example, the emergency situation may be determined in response to a suspicious object detection, such as a sharp object or other potentially threaten object in the vehicle cabin.

[0103] At 804, in response to determining an emergency situation is indicated, the method 800 may include automatically generating a notification and displaying the notification to the driver via an SMS or an icon in an instrument cluster. For example, in response to the ADAS detecting a sharp object or other potentially threaten object in the vehicle cabin, the ADAS may generate the notification. In some examples, automatic notification generation may be determined based on the detected situation, and may not be generated in some situations.

[0104] At 805, the method 800 may include automatically generating an emergency service request including the vehicle number (e.g., license plate number, registration, etc.), the location (e.g., GPS coordinates), and a picture, and transmitting the emergency service requestto an appropriate service, such as police, fire, or medical assistance. For example, in response to the ADAS detecting the vehicle is not upright, the ADAS may automatically generate the emergency service request. In some examples, automatic service request generation may be determined based on the detected situation, and may not be generated in some situations.

[0105] In response to determining an emergency situation is not detected, the method 800 may proceed to 806. At 806, the method 800 may include capturing video footage of a user via a user-facing camera at a second framerate. In one or more examples, the camera may be the NIR camera described above with reference to the method 700. The second framerate may be a greater framerate than the first framerate, such as 70, 80, 100 FPS, or another value that may be calibrated for analyzing facial movement data to determine user requests.

[0106] At 808, the method 800 may include determining first order facial movement for each frame of the video footage. In one example, the first order facial movement may include eye gaze vectors and eyelid openness levels for each frame of acquired image data, as described above with reference to the method 700. First order facial movement may additionally, or alternatively, include lip shapes and lip positions for each frame of acquired image data. Determining hp shapes and lip positions may involve processing the captured images to extract geometric and numeric mouth and / or lip shape information. For example, the method 800 may include performing operations such as facebox detection, mouthbox detection, head position detection, lip position detection, tongue detection (position), teeth detection (position), and the determination of the mouth and / or hp shape itself from the distance between the lips, the lips and the tongue, the tongue and the teeth, etc.

[0107] At 810, the method 800 may include determining a plurality of second order facial movement metrics from the first order facial movements over a pre-determined time window. In one example, the pre-determined time window may be an adjustable time window, which may be adjusted based on a facial movement metric of interest, such as eye movement data and hp movement data, and which facial movement interpretation model is to be used in analysis of the facial movement data. For example, the adjustable time window may be adjusted to a first time window when analyzing eye movement data, and adjusted to a second time window when analyzing lip movement data. The plurality of second order facial movement metrics may be determined from eye gaze vectors and eyelid openness for eye movement data, and from hp shape and hp position for lip movement data. The second order eye movement metrics may include one or more of the metrics described with reference to the method 700 including eye behavioral patterns in time, such as fixations, saccades, blinks, and long closures. The second order eye movement metrics may additionally, or alternatively, include eye behaviorsthat correspond to pre-determined messages in Morse code, including short blinks and long blinks, and patterns of blinks, which may be defined based on duration and degree of eyelid closure or other metrics. The second order lip movement metrics may include lip movement behaviors that are involved in speech production such as the shape formed by the lips, a distance between lips, lips touching, tongue contact with lips, tongue contact teeth, etc., and may further include the length of the behavior. Lip movement behaviors may correspond to discrete speech units, such a single syllable or a single syllable word, or compound speech units, such as a multiple syllable word or phrase.

[0108] In one or more examples, the method 800 may include sub-processes for determining the plurality of second order facial movement metrics from the first order facial movements. As one non-limiting example, a sub-process for detecting long blinks and short blinks may include receiving a time sequence of eyelid openness levels and calculating differences between adjacent eyelid openness levels for each frame. The process may include identifying an initial closing point or onset of a potential blink, and identifying a physiological closing point, which is a point in a blink or long closure where the eyelid is in a substantially closed position. The process may include identifying the start of the eye opening, and identifying the end of the eye opening. The process may include determining whether the blink satisfies a blink criterion. For example, a blink may have a first blink criterion, such as the blink event falling within a range defined by minimum and maximum short blink duration thresholds, such as 5 and 50 frames. If the blink event falls within the minimum and maximum short blink duration, the blink may be stored as a short blink. If the blink event exceeds the maximum short blink duration, the blink may be stored as a long blink. Otherwise, the eye movement may be discarded, or evaluated through another similar process to determine other eye movement behaviors such as saccades or fixations. In one or more examples, the method 700 may implement similar sub-processes for determining second order eye movement metrics.

[0109] As another non-limiting example, a sub-process for detecting discrete speech units may include receiving a time sequence of lip positions and calculating differences between adjacent lip positions for each frame. The process may include identifying points of lip contraction, or extension, which may indicate the onset of a speech unit or syllable. The process may include identifying the start of a lip extension from a point of contraction, and identify ing the end of the lip extension, which may indicate the end of the speech unit. The process may include determining whether the discrete speech unit satisfies a speech unit criterion. The speech unit criterion may be based on the shape, such as a two-dimensional representation of the speech unit at extension (or contraction), falling within a threshold rangedefined by a minimum dimension and a maximum dimension. The threshold range may be calibrated to a speech unit or syllable. For example.CLS” may have maximum dimension and a minimum dimension, which may be different than the maximum dimension and the minimum dimension calibrated for “O”. Additionally, or alternatively, the speech unit criterion may be based on a duration of the speech unit, such as the speech event falling within a range defined by minimum and maximum duration thresholds for the speech unit or syllable, such as 5 and 100 frames. If the speech event satisfies the speech unit criterion, the speech unit may be stored. Otherwise, the speech unit may be discarded.

[0110] At 812, the method 800 may include transforming the plurality' of second order facial movement metrics into a machine readable representation for the pre-determined time window. As described above with reference to the method 800, the transformation of second order eye movement metrics may include converting the discrete sequence of blinks, saccades, and fixations into a numerical metric that defines the "amount" of patterns that occurred during the time window. In one example, the machine readable representation may include characteristics such as the number of blinks, the duration of the blinks (e.g., long blink, short blink), and the pattern of long blinks and short blinks. The transformation of second order lip movement metrics may include converting the sequence of discrete speech units into a computer readable metric that defines the patterns of lip shapes and positions corresponding to speech production that occurred during the time window. In some examples, the window may be sliding, allowing for the evaluation of facial movement metrics within overlapping time window.[OHl] At 814, the method 800 may include determining one or more user requests for the user based on the machine readable representation of the plurality' of second order facial movement metrics. In one example, the determination is facilitated by a machine learning model that is trained to detect user requests, such as the code reading model 322 and the code reading models 420 in FIGS. 3-4, respectively. In some examples, the machine learning model may be trained on machine readable representations derived from facial movement data corresponding to pre-determined commands. The machine learning model based be trained to detect from eye movement data one or more of a plurality of messages in Morse code corresponding to user requests, and to detect from lip movement data one or more of a plurality of lip reading messages corresponding to user requests. For example, the model may be trained to detect one or both of an eye blinking pattern or lip movement pattern that corresponds to "SOS". In additional, or alternative, examples, the machine learning model may include deep neural networks that learn correlations between the metrics and user-specific commands.

[0112] At 816, the method 800 may include outputting the one or more determined user requests. In one example, outputting may include generating a control signal based on the determined user request and transmitting the control signal to the central processing unit of the vehicle for real-time intervention.

[0113] At 818, the method 800 may include executing one or more use assistance operations based on the determined user requests. In one example, the one or more user assistance operations may include at least one of activating a vehicle function, ceasing a vehicle function, adjusting a vehicle operating condition, and requesting emergency assistance.

[0114] At 820, the method may include determining whether the indication of an emergency situation is resolved. As one example, resolution may be determined in response to the ADAS receiving an indication via the infotainment system, such as in response to the driver giving a command. For example, the driver may give a verbal command, an eye-blinked command, a lip reading command, or by hitting a driver assistance button. In another example, resolution may be determined in response to the ADAS receiving an indication from an emergency service communication system, e.g., that fire, medical, or police service has been dispatched.

[0115] In some examples, the method 800 may proceed to 806 after one or both of automatically generating a driver notification at 804 and automatically generating an emergency service request at 805. Alternatively, in some examples, the method 800 may proceed directly to 806 without determining whether an emergency situation is indicated, e.g., regardless of detection of a distress signal. For example, an emergency situation may be indicated upon analysis of facial movement behavior corresponding to user requests. For example, an emergency situation may be indicated in response to the ADAS detecting an ‘‘SOS'’ message from the analysis of the facial movement data.

[0116] In this way, the systems and methods for advanced driver assistance systems monitor driver behavior and assist with the safe operation of a vehicle. The approach enables the ADAS to monitor a vehicle cabin and its occupants in at least two monitoring modes, which allows the detection risks associated with impaired driving due to factors such as drowsiness and stress, and assistance in situations where the driver or passenger cannot move his or her hands, such as in an emergency situation or an accident, where the driver may be unable to request emergency assistance. The disclosed driver assistance approaches may reduce delay in the driver or passenger obtaining a medical or emergency response. By incorporating logic that controls which monitoring mode is commanded, the approach may demand no more processing power than a system that monitors user facial movement in one mode only.

[0117] The disclosure also provides support for a method for an advanced driver alert system for a vehicle comprising: monitoring, via a user-facing camera, facial movement of a user over time, determining a plurality of facial movement metrics based on the facial movement over an adjustable time window, transforming the plurality of facial movement metrics into a machine readable representation of the adjustable time window7, determining, via a code reading model, one or more user requests based on the machine readable representation of the plurality of facial movement metrics, and executing one or more user assistance operations based on the determined user requests. In a first example of the method, the method further comprises: adjusting the adjustable time window based on a facial movement metric of interest. In a second example of the method, optionally including the first example, the facial movement comprises at least one of eye movement and lip movement. In a third example of the method, optionally including one or both of the first and second examples, the plurality of facial movement metrics comprises one or more of blink number, blink pattern, blink duration, lip shapes, lip patterns, and lip shape duration. In a fourth example of the method, optionally including one or more or each of the first through third examples, the method further comprises: training the code reading model based on machine readable representations derived from at least one of eye movement data to detect one or more of a plurality of messages in Morse code corresponding to user requests, and lip movement data to detect one or more of a plurality of lip reading messages corresponding to user requests. In a fifth example of the method, optionally including one or more or each of the first through fourth examples, executing one or more user assistance operations based on the determined user requests comprises at least one of activating a vehicle function, ceasing a vehicle function, adjusting a vehicle operating condition, and requesting emergency assistance. In a sixth example of the method, optionally including one or more or each of the first through fifth examples, the method further comprises : training the code reading model based on machine readable representations derived from situational data corresponding to potentially dangerous or threatening situations. In a seventh example of the method, optionally including one or more or each of the first through sixth examples, the method further comprises: monitoring, via the user-facing camera, a vehicle cabin, determining, via the code reading model, a suspicious situation, in response to determining a first situation, automatically generating a user notification, and in response to determining a second situation, automatically generating an emergency assistance request comprising a vehicle license number, location, and image. In a eighth example of the method, optionally including one or more or each of the first through seventh examples, the method further comprises: activating the monitoring in response to receiving an indication one or moreof greater than a first threshold psychophysiological state of the user, a vehicle impact greater than a vehicle impact threshold, a vehicle stop followed by a passenger entry to the vehicle occurring within a duration threshold, and a suspicious object detection. In a ninth example of the method, optionally including one or more or each of the first through eighth examples, the method further comprises: capturing user-specific facial movement data corresponding to a plurality of user requests, and training the code reading model based on machine readable representations derived from the user-specific facial movement data.

[0118] The disclosure also provides support for a method for an advanced driver alert system for a vehicle comprising: operating in a first monitoring mode in response to a first set of conditions and operating in a second monitoring mode in response to a second set of conditions, wherein the first monitoring mode comprises monitoring, via a user-facing camera, eye movement of a user over time and predicting, via a psychophysiological state prediction model, one or more psychophysiological states of the user based on the eye movement, and wherein the second monitoring mode comprises monitoring, via the user-facing camera, at least one of eye movement and lip movement of the user over time, determining, via a code reading model, one or more user requests based on the at least one of eye movement and lip movement, and executing one or more user assistance operations based on the determined user requests. In a first example of the method, the second set of conditions comprises a psychophysiological state of the user greater than a first threshold, and wherein the first set of conditions comprises the psychophysiological state of the user less than or equal to the first threshold. In a second example of the method, optionally including the first example the second monitoring mode further comprising determining a plurality of at least one of eye movement metrics and lip movement metrics respectively based on the at least one of eye movement and lip movement over an adjustable time window, transforming the plurality of the at least one of eye movement metrics and lip movement metrics into a machine readable representation of the adjustable time window, and determining, via the code reading model, one or more user requests based on the machine readable representation of the at least one of the eye movement metrics and lip movement metrics. In a third example of the method, optionally including one or both of the first and second examples, executing one or more user assistance operations based on the determined user requests comprises at least one of activating a vehicle function, ceasing a vehicle function, adjusting a vehicle operating condition, and requesting emergency assistance. In a fourth example of the method, optionally including one or more or each of the first through third examples, the method further comprises: training the code reading model based on machine readable representations derived from at least one of eye movement data to detect oneor more of a plurality of messages in Morse code corresponding to user requests, and lip movement data to detect one or more of a plurality of lip reading messages corresponding to user requests.

[0119] The disclosure also provides support for an advanced driver alert system (ADAS) for a vehicle, comprising: an infotainment system configured to execute a plurality of user assistance operations, a camera configured to capture a sequence of facial movements over time, a non-transitory memory storing instructions, a psychophysiological state prediction model, and a code reading model, and a processor communicably coupled to the camera, the infotainment system, and the non-transitory memory, the processor, when executing the instructions, configured to: monitor in a first operating mode, via the camera, eye movement of a user over time and predict, via the psychophysiological state prediction model, one or more psychophysiological states of the user based on the eye movement, and in response to an indication of greater than a first threshold psychophysiological state, monitor in a second monitoring mode, via the camera, at least one of eye movement and lip movement of the user over time, determine, via the code reading model, one or more user requests based on the at least one of eye movement and lip movement, and execute one or more user assistance operations based on the determined user requests. In a first example of the system, the code reading model comprises a machine learning model configured to detect one or more of a plurality of messages in Morse code corresponding to user requests. In a second example of the system, optionally including the first example, the code reading model comprises a machine learning model configured to detect one or more of a plurality of lip reading messages corresponding to user requests. In a third example of the system, optionally including one or both of the first and second examples, the code reading model comprises a machine learning model trained to detect user-specific facial movement corresponding to a plurality of user requests based on machine readable representations derived from user-specific facial movement data. In a fourth example of the system, optionally including one or more or each of the first through third examples, the one or more user assistance operations comprises at least one of activating a vehicle function, ceasing a vehicle function, adjusting a vehicle operating condition, and requesting emergency assistance.

[0120] The description of embodiments has been presented for purposes of illustration and description. Suitable modifications and variations to the embodiments may be performed in light of the above description or may be acquired from practicing the methods. For example, unless otherwise noted, one or more of the described methods may be performed by a suitable device and / or combination of devices, such as the infotainment system 109 described withreference to FIGS. 1 and 2 and user 302 described with reference to FIG. 3. The methods may be performed by executing stored instructions with one or more logic devices (e.g., processors) in combination with one or more additional hardware elements, such as storage devices, memory, hardware network interfaces / antennas, switches, actuators, clock circuits, etc. The described methods and associated actions may also be performed in various orders in addition to the order described in this application, in parallel, and / or simultaneously. The described systems are exemplary in nature, and may include additional elements and / or omit elements. The subject matter of the present disclosure includes all novel and non-obvious combinations and sub-combinations of the various systems and configurations, and other features, functions, and / or properties disclosed.

[0121] As used in this application, an element or step recited in the singular and proceeded with the word '‘a” or “an” should be understood as not excluding plural of said elements or steps, unless such exclusion is stated. Furthermore, references to “one embodiment” or “one example” of the present disclosure are not intended to be interpreted as excluding the existence of additional embodiments that also incorporate the recited features. The terms “first,” “second; and “third.” etc. are used merely as labels, and are not intended to impose numerical requirements or a particular positional order on their objects. The following claims particularly point out subject matter from the above disclosure that is regarded as novel and non-obvious.

Claims

CLAIMS:1 . A method for an advanced driver alert system for a vehicle comprising: monitoring, via a user-facing camera, facial movement of a user over time; determining a plurality' of facial movement metrics based on the facial movement over an adj ustable time window; transforming the plurality of facial movement metrics into a machine readable representation of the adjustable time window; determining, via a code reading model, one or more user requests based on the machine readable representation of the plurality of facial movement metrics; and executing one or more user assistance operations based on the determined user requests.

2. The method of claim 1, further comprising adjusting the adjustable time window based on a facial movement metric of interest.

3. The method of claim 1. wherein the facial movement comprises at least one of eye movement and lip movement.

4. The method of claim 1, wherein the plurality of facial movement metrics comprises one or more of blink number, blink pattern, blink duration, lip shapes, lip patterns, and lip shape duration.

5. The method of claim 1, further comprising training the code reading model based on machine readable representations derived from at least one of eye movement data to detect one or more of a plurality of messages in Morse code corresponding to user requests, and lip movement data to detect one or more of a plurality^ of lip reading messages corresponding to user requests.

6. The method of claim 1. wherein executing one or more user assistance operations based on the determined user requests comprises at least one of activating a vehicle function, ceasing a vehicle function, adjusting a vehicle operating condition, and requesting emergency assistance.

7. The method of claim 1, further comprising training the code reading model based on machine readable representations derived from situational data corresponding to potentially dangerous or threatening situations.

8. The method of claim 1, further comprising: monitoring, via the user-facing camera, a vehicle cabin; determining, via the code reading model, a suspicious situation; in response to determining a first situation, automatically generating a user notification; and in response to determining a second situation, automatically generating an emergency assistance request comprising a vehicle license number, location, and image.

9. The method of claim 1, further comprising activating the monitoring in response to receiving an indication one or more of greater than a first threshold psychophysiological state of the user, a vehicle impact greater than a vehicle impact threshold, a vehicle stop followed by a passenger entry to the vehicle occurring within a duration threshold, and a suspicious object detection.

10. The method of claim 1, further comprising capturing user-specific facial movement data corresponding to a plurality of user requests, and training the code reading model based on machine readable representations derived from the user-specific facial movement data.

11. A method for an advanced driver alert system for a vehicle comprising: operating in a first monitoring mode in response to a first set of conditions and operating in a second monitoring mode in response to a second set of conditions, wherein the first monitoring mode comprises monitoring, via a user-facing camera, eye movement of a user over time and predicting, via a psychophysiological state prediction model, one or more psychophysiological states of the user based on the eye movement, and wherein the second monitoring mode comprises monitoring, via the user-facing camera, at least one of eye movement and lip movement of the user over time, determining, via a code reading model, one or more user requests based on the at least one of eye movement and lip movement, and executing one or more user assistance operations based on the determined user requests.

12. The method of claim 11, wherein the second set of conditions comprises a psychophysiological state of the user greater than a first threshold, and wherein the first set of conditions comprises the psychophysiological state of the user less than or equal to the first threshold.

13. The method of claim 11, the second monitoring mode further comprising determining a plurality of at least one of eye movement metrics and lip movement metrics respectively based on the at least one of eye movement and lip movement over an adjustable time window, transforming the plurality of the at least one of eye movement metrics and lip movement metrics into a machine readable representation of the adjustable time window, and determining, via the code reading model, one or more user requests based on the machine readable representation of the at least one of the eye movement metrics and lip movement metrics.

14. The method of claim 11, wherein executing one or more user assistance operations based on the determined user requests comprises at least one of activating a vehicle function, ceasing a vehicle function, adjusting a vehicle operating condition, and requesting emergency assistance.

15. The method of claim 11, further comprising training the code reading model based on machine readable representations derived from at least one of eye movement data to detect one or more of a plurality of messages in Morse code corresponding to user requests, and lip movement data to detect one or more of a plurality of lip reading messages corresponding to user requests.

16. An advanced driver alert system (ADAS) for a vehicle, comprising: an infotainment system configured to execute a plurality' of user assistance operations; a camera configured to capture a sequence of facial movements over time; a non-transitory memory storing instructions, a psychophysiological state prediction model, and a code reading model; and a processor communicably coupled to the camera, the infotainment system, and the non-transitory memory7, the processor, when executing the instructions, configured to: monitor in a first operating mode, via the camera, eye movement of a user over time and predict, via the psychophysiological state prediction model, one or more psychophysiological states of the user based on the eye movement, andin response to an indication of greater than a first threshold psychophysiological state, monitor in a second monitoring mode, via the camera, at least one of eye movement and lip movement of the user over time, determine, via the code reading model, one or more user requests based on the at least one of eye movement and lip movement, and execute one or more user assistance operations based on the determined user requests.

17. The ADAS of claim 16, wherein the code reading model comprises a machine learning model configured to detect one or more of a pl urality of messages in Morse code corresponding to user requests.

18. The ADAS of claim 16, wherein the code reading model comprises a machine learning model configured to detect one or more of a plurality of lip reading messages corresponding to user requests.

19. The ADAS of claim 16, wherein the code reading model comprises a machine learning model trained to detect user-specific facial movement corresponding to a plurality of user requests based on machine readable representations derived from user-specific facial movement data.

20. The ADAS of claim 16, wherein the one or more user assistance operations comprises at least one of activating a vehicle function, ceasing a vehicle function, adjusting a vehicle operating condition, and requesting emergency assistance.

Images