Methods and systems for managing cognitive load of vehicle drivers

Abstract

A cognitive load management system may continuously monitor a cognitive state of a driver of a vehicle, via a driver monitoring system (DMS). When a detected or estimated cognitive load of the driver increases above a first threshold cognitive load, the cognitive load management system may adjust one or more controls of the vehicle to reduce a stress of the driver. When the detected or estimated cognitive load of the driver decreases below a second threshold cognitive load, at which drowsiness or disengagement may impair a performance of the driver, a cognitive stimulator component of the system may engage the driver in conversation using a generative AI model. In this way, different strategies may be employed to address both high and low cognitive loads.

Description

Attorney Docket No. P240098WO METHODS AND SYSTEMS FOR MANAGING COGNITIVE LOAD OF VEHICLE DRIVERS FIELD

[0001] The disclosure relates generally to increasing the performance of driver monitoring systems of vehicles. BACKGROUND

[0002] Current driver monitoring system (DMS) may be used to estimate a physiological state (such as a drowsiness or a distraction), of a driver of a vehicle. A DMS may be based on sensors including contactless sensors that analyze various aspects or parameters of driver behavior, such as, for example, head movement, eye gaze, and / or other similar aspects or parameters of driver behavior that may indicate or relate to the physiological state of the driver. For example, a first set of patterns in head movement and / or eye gaze data may indicate that the driver may be drowsy, a second set of patterns in the head movement and / or eye gaze data may indicate that the driver may be distracted, and so forth.

[0003] Next generation DMS systems extend the functionality in considering cognitive aspects or moods of the driver based on sensor data. If the physiological state of the driver can be accurately assessed or predicted, under certain circumstances, a controller of the vehicle may intervene to assist the driver in operating the vehicle. For example, if the DMS detects that the driver is drowsy, a controller of the vehicle may alert the driver via an audio recording, adjust a responsiveness of an accelerator pedal or brake pedal of the vehicle, issue a visual or audio notification, adjust a lighting of the vehicle, adjust a heating, ventilation and air conditioning (HVAC) setting of the vehicle, and / or adjust a different control of the vehicle.

[0004] However, the interventions may be generated at a point where the driver is already showing signs of drowsiness or impairment, and may not be generated early enough to improve driving. Additionally, the interventions may alert the driver to signs of impairment, but may not help the driver maintain a level of cognitive awareness and engagement over a period of time during which the driver is operating the vehicle. SUMMARY

[0005] In various embodiments, the issues described above may be addressed by a method, comprising estimating a cognitive load of a driver of a vehicle based on information received from a driver monitoring system (DMS) of the vehicle; in response to the estimated cognitiveAttorney Docket No. P240098WO load being above a first threshold, adjusting one or more cabin controls of the vehicle to reduce the cognitive load; and in response to the cognitive load being below a second threshold, engaging the driver in conversation to increase the cognitive load, where the conversation is generated using a generative artificial intelligence (AI) model.

[0006] It should be understood that the summary above is provided to introduce in simplified form a selection of concepts that are further described in the detailed description. It is not meant to identify key or essential features of the claimed subject matter, the scope of which is defined uniquely by the claims that follow the detailed description. Furthermore, the claimed subject matter is not limited to implementations that solve any disadvantages noted above or in any part of this disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The disclosure may be better understood from reading the following description of non-limiting embodiments, with reference to the attached drawings, wherein below:

[0008] FIG.1 is a schematic block diagram of a vehicle control system, in accordance with one or more embodiments of the present disclosure;

[0009] FIG.2 is a schematic block diagram that shows an exemplary flow of data through components of a cognitive load management system of a vehicle, in accordance with one or more embodiments of the present disclosure;

[0010] FIG. 3 is a flowchart illustrating an exemplary high-level method for initiating a cognitive load management system of a vehicle, in accordance with one or more embodiments of the present disclosure;

[0011] FIG.4 is a flowchart illustrating a method for engaging a driver of a vehicle in AI- generated conversation based on a predicted cognitive load of the driver, in accordance with one or more embodiments of the present disclosure;

[0012] FIG. 5 is a diagram showing an interaction of a driver with a cognitive stimulator component of the cognitive load management system, in accordance with one or more embodiments of the present disclosure;

[0013] FIG.6 shows an exemplary dashboard of a vehicle including a plurality of controls, in accordance with one or more embodiments of the present disclosure; and

[0014] FIG.7 is a schematic block diagram that shows an in-vehicle computing system and a control system of a vehicle, in accordance with one or more embodiments of the present disclosure.Attorney Docket No. P240098WO DETAILED DESCRIPTION

[0015] The following detailed description relates to managing a cognitive load of a driver of a vehicle to increase a safety of the driver and the vehicle. Cognitive load is related to whether the user is occupied with a first task or small set of tasks, e.g., driving a vehicle, or whether the user is simultaneously occupied with other tasks in addition to the first task or small set of tasks. Thus, the cognitive load of the driver may increase when there are additional tasks to perform, and may decrease when there are no additional tasks to perform. When the cognitive load is high (e.g., above a first threshold), stress may be generated which may decrease an ability of the driver to make decisions, perform driving actions, and / or react to stimuli, which may decrease the safety of the driver. Conversely, when the cognitive load is low (e.g., below a second threshold) the driver may become bored or drowsy, which may increase a negligence or a disengagement of the driver, also decreasing the safety of the driver.

[0016] The disclosure is based on the principle that the cognitive load of the driver can be determined using externally observable criteria captured in images via a camera of the vehicle, for example, a driver monitoring system (DMS) camera of the vehicle. For example, the driver may exhibit a first set of postural or facial expression characteristics when experiencing high cognitive load, such as when executing a cognitively demanding task, and a second set of postural or facial expression characteristics when experiencing a low cognitive load, e.g., when relaxed. An artificial intelligence (AI) model, such as a convolutional neural network (CNN), may be trained to estimate or predict the cognitive load based on the images. The AI model may classify the cognitive load, and output a score representing the cognitive load of the driver.

[0017] The inventors herein have recognized that managing the cognitive load of the driver may include both reducing the cognitive load of the driver when the cognitive load is high, and increasing the cognitive load of the driver when the cognitive load is low. Various approaches have been taken towards reducing the high cognitive load of the driver. For example, when the cognitive load of the driver exceeds a threshold score, in-cabin environmental controls including lighting, temperature, music, etc. may be adjusted to reduce a stress of the driver. However, few solutions to driver disengagement due to low cognitive loads of the driver have been proposed.

[0018] To address this issue, systems and methods are disclosed herein for selectively engaging drivers when low cognitive load is detected, using gamification and conversational generative AI. In recent years, generative AI has seen significant advancements due to progress in natural language processing (NLP) and machine learning (ML) techniques. As used herein, conversational generative AI refers to the generation of a series of conversation segments inAttorney Docket No. P240098WO any media between a generative AI model, such as a public or private large language model (LLM), and a human (e.g., a driver) in an exchange of natural language, data and / or information, that adheres to linguistic rules for syntax and semantics such as informality, ambiguity, extension, evolution, self-reference, and contradiction. The data and / or information exchanged in the conversation may also include graphical data (e.g., pictures, sketches, schematics) as well as gestures, pantomime, and other encoding means such as chemical, tactile, and other sensory modalities.

[0019] Conversational generative AI is routinely performed by “chatbots” developed to converse with a human (or humans) in many disciplines. Digital voice assistants, smart translation devices or software such as Amazon Alexa, Apple Siri, and Google Home inherently contain a chatbot capability as their main interface with the user. Many different programmed algorithms, and enormous corpuses of human-created and juried content, have been developed to enable the chatbot to maintain a human-like conversation. The sophistication of chatbot implementations ranges from simple declarative programs to elaborately trained neural networks.

[0020] A cognitive load management system may continuously monitor a cognitive state of the driver, for example, via a driver monitoring system (DMS). When a detected or estimated cognitive load of the driver decreases below a first threshold cognitive load, at which drowsiness or disengagement may impair a performance of the driver, a cognitive stimulator component of the system may engage the driver in conversation using a generative AI model. When the detected or estimated cognitive load of the driver increases above a second threshold cognitive load, the cognitive load management system may adjust one or more controls of the vehicle to reduce a stress of the driver. In this way, different strategies may be employed to address both high and low cognitive loads.

[0021] For example, the cognitive stimulator may ask the driver questions via an audio system of the vehicle, and the driver may respond via a microphone of the vehicle. The questions generated by the cognitive stimulator component may vary. Some questions may be based on topics of interest to the driver, which may be determined via a companion app installed on a smart phone of the driver, in some embodiments. Other questions may be directed to elements of an environment of the driver or vehicle. For example, the cognitive stimulator may ask the driver to estimate a distance between the vehicle and a preceding vehicle. By engaging the driver in conversation, the cognitive stimulator may increase the driver’s cognitive load, preventing the drowsiness, and increasing the safety of the driver.Attorney Docket No. P240098WO

[0022] Referring now to FIG. 1, a simplified vehicle control system 100 of a vehicle is shown, including a controller 102, a plurality of sensors 120, a navigational system 154, and a plurality of output devices 130. Controller 102 may include a processor 104, which may execute instructions stored on a memory 106 to generate output signals, such as audio or video signals, at output devices 130 based at least partly on an output of sensors 120.

[0023] As discussed herein, the memory 106 may include any non-transitory computer readable medium in which programming instructions are stored. For the purposes of this disclosure, the term “tangible computer readable medium” is expressly defined to include any type of computer readable storage. The example methods and systems may be implemented using coded instruction (e.g., computer readable instructions) stored on a non-transitory computer readable medium such as a flash memory, a read-only memory (ROM), a random- access memory (RAM), a cache, or any other storage media in which information is stored for any duration (e.g. for extended period time periods, permanently, brief instances, for temporarily buffering, and / or for caching of the information). Computer memory of computer readable storage mediums as referenced herein may include volatile and non-volatile or removable and non-removable media for a storage of electronic-formatted information such as computer readable program instructions or modules of computer readable program instructions, data, and so on that may be stand-alone or as part of a computing device. Examples of computer memory may include any other medium which can be used to store the desired electronic format of information and which can be accessed by the processor or processors or at least a portion of a computing device.

[0024] The one or more sensors 120 of the vehicle may include one or more vehicle sensors 150. Data outputted by vehicle sensors 150 may be an input into controller 102. Vehicle sensors 150 may include, for example, engine speed and / or wheel speed sensors, which may indicate a speed of the vehicle or used to calculate acceleration of the vehicle. Vehicle sensors 150 may also include one or more in-cabin sensors arranged within a cabin of the vehicle. The one or more in cabin sensors may include one or more cameras, such as a dashboard camera, which may be used to collect images of the driver and / or passengers of the vehicle for further processing. The one or more in-cabin sensors may include one or more microphones 170 arranged on a dashboard of the vehicle and / or a different part of the cabin of the vehicle, which may be used to receive audio input from the driver and / or to determine a level of noise within the cabin and / or generate contextual data based on audio signals detected within the cabin. The one or more in-cabin sensors may include one or more seats sensors of the vehicle, which may be used to determine a seat occupancy of the vehicle and / or identify one or more passengersAttorney Docket No. P240098WO and / or types of passengers. The in-cabin sensors and environment of the vehicle is described in greater detail in reference to FIGS.6 and 7 below.

[0025] The one or more sensors 120 of the vehicle may include one or more external sensors 152, and sensor data of external sensors 152 may be an input into controller 102. External sensors 152 may include, for example, one or more external cameras, such as a front end camera and a rear end camera; radar, lidar, and / or proximity sensors of the vehicle, which may detect a proximity of objects (e.g., other vehicles) to the vehicle; sensors of a windshield wiper, lights, and / or a sunroof, which may be used to determine an environmental context of the vehicle; and / or sensors of one or more indicator lights to the vehicle, which may be used to determine a traffic scenario of the vehicle.

[0026] The one or more sensors 120 may include a DMS 171. DMS 171 may monitor the driver to detect or measure aspects of a cognitive state of the driver, for example, via a dashboard camera of the vehicle, or via one or more sensors arranged in the cabin of the vehicle. Biometric data of the driver (e.g., vital signs, galvanic skin response, and so on) may be collected from a sensor of a driver’s seat of the vehicle, or a sensor on a steering wheel of the vehicle, or a different sensor in the cabin. DMS 171 may analyze dashboard camera data, biometric data, and other data of the driver to generate an output. In various embodiments, the output of DMS 171 may be a detected or predicted cognitive state of the driver. In particular, DMS 171 may include a cognitive load estimator 172, which may output an estimated cognitive load of the driver based on images acquired via DMS 171, and / or other sensor data of the sensors 120, which may be correlated with a stress level of the driver, a level of distraction of the driver, and / or a level of drowsiness of the driver. In some embodiments, cognitive load estimator 172, or an additional cognitive load estimator, may be a separate component outside of DMS 171, where data of DMS 171 (e.g., image data, biometric data, interpreted data, etc.) is outputted to the separate component, and the separate component estimates the cognitive load of the driver. In various embodiments, cognitive load estimator 172 may output a cognitive load score based on the data and / or the output of DMS 171, where the cognitive load score represents a degree of cognitive load of the driver. For example, the cognitive load score may include a value from one to 10, where one indicates a low cognitive load, and 10 indicates a high cognitive load.

[0027] Controller 102 includes a cognitive load management system 108, which may monitor the cognitive load of the driver received from the cognitive load estimator 172. Cognitive load management system 108 may include a cognitive threshold detector 110, which may detect when a cognitive load estimated by cognitive load estimator 172 decreases belowAttorney Docket No. P240098WO a first threshold cognitive load (e.g., a lower cognitive load) or increases above a second threshold cognitive load (e.g., a higher cognitive load). Cognitive load management system 108 includes a cognitive stimulator 112, which may stimulate or engage the driver when the cognitive load of the driver decreases below the first threshold cognitive load. As described in greater detail below, cognitive stimulator 112 may stimulate the driver by engaging the driver in conversation using a generative AI model 114. Cognitive stimulator 112 may also include a text-to-voice converter 116, which may convert an output of generative AI model 114 to an audio stream for engaging the driver. Cognitive load management system 108 may also take as input route data of the vehicle received from navigational system 154, such as, for example, a type of road or roads on a route of the vehicle, an estimated time to reach a destination of the vehicle, and / or a driving environment of the vehicle (e.g., rural, city, highway, etc.).

[0028] Cognitive stimulator 112 may engage the driver in conversation based on interests of the driver stored in an interest profile 122. As described in greater detail below, interest profile 122 may be generated by an interest model 118, based on data extracted from a smart phone of the driver. Interest model 118 may be an AI model, such as a rules-based model generated by human experts, or a statistical model, or a classification or clustering model, or a different kind of model. In some example, interest model 118 may be a machine learning (ML) model trained on training data extracted from a plurality of smart phones of a plurality of users. Cognitive stimulator 112 may also include a set of cognitive load mappings 124, which may be stored in a lookup table or database stored in memory 106. Cognitive load mappings 124 may map various topics or types of topics of interest to different cognitive loads. For example, cognitive stimulator 112 may engage the driver in conversation using generative AI model 114 based on a cognitive load score of the driver, and cognitive stimulator 112 may select a topic of interest to the driver based on the cognitive load score. That is, if the cognitive load score is higher, cognitive stimulator 112 may prompt generative AI model 114 to initiate a conversation on a first topic of interest associated with a lower cognitive load, such as a sport the driver is interested in. (If the cognitive load is above a threshold cognitive load, cognitive stimulator 112 may not be used.) If the cognitive load score is lower, cognitive stimulator 112 may prompt generative AI model 114 to initiate a conversation on a second topic of interest associated with a higher cognitive load, such as a topic in a news feed of the driver, where the second topic of interest may cause the driver to engage in the conversation at a higher cognitive level than the first topic of interest. Cognitive load mappings 124 may be generated by human experts, in some embodiments. In other embodiments, cognitive load mappings 124 may be determined by a model, such as a statistical model or an AI model.Attorney Docket No. P240098WO

[0029] While cognitive stimulator 112 is depicted as a component of cognitive load management system 108 of controller 102, it should be appreciated that in various embodiments, one or more components of cognitive stimulator 112 may be located at a cloud- based server 156, where the one or more components may take advantage of comparatively greater processing and memory resources available at cloud-based server 156 with respect to controller 102. In other words, components of controller 102 may interface and interact with the driver, and transmit information to cloud-based server 156 (e.g., via network 160) to be further processed. After the information has been further processed at cloud-based server 156, the processed information may be transmitted from cloud-based server 156 to cognitive load management system 108 for further interaction with the driver. In this way, processing tasks inherent to generating and maintaining a conversation with the driver may be advantageously shifted or balanced between cloud-based server 156 and cognitive load management system 108 to minimize a delay in responding to the driver.

[0030] Output devices 130 include at least one or more speakers 132, a display of an in- vehicle infotainment (IVI) system 134, and a plurality of cabin controls 136. Output devices 130 may be relied on by cognitive load management system 108 to intervene when the cognitive load of the driver decreases below the first threshold cognitive load or increases above the second threshold cognitive load. For example, when the cognitive load of the driver decreases below the first threshold cognitive load, cognitive stimulator 112 may engage the driver in conversation via one or more speakers 132. The driver may respond via microphone 170. In some embodiments, cognitive stimulator 112 may engage the driver visually via IVI system 134 in addition to or as an alternative to engaging the driver in conversation, as described in greater detail below. In some embodiments, the one or more speakers 132 may include a personal listening device of the driver, such as headphones or ear buds of the driver, where the conversation may be generated via an audio stream transmitted to the personal listening device via a wireless network 160 of the vehicle. Alternatively, when the cognitive load of the driver increases above the second threshold cognitive load, cognitive load management system 108 may actuate one or more cabin controls 136, such as a temperature control, an ambient lighting control, etc., to reduce the driver’s cognitive load.

[0031] As an example, the driver may operate the vehicle when drowsy. As a result of being drowsy, the driver may exhibit a pattern of head movements (such as a periodic dipping of the driver’s head). DMS 171 may detect the pattern of movements, and may codify the pattern of movements as indicative of a low cognitive load. Cognitive load estimator 172 may output a cognitive load score of the driver, which is received at cognitive load managementAttorney Docket No. P240098WO system 108. When the cognitive load score is received at cognitive load management system 108, cognitive threshold detector 110 may determine that the cognitive load score is below the first cognitive load threshold. As a result of being below the first cognitive load threshold, cognitive stimulator 112 may initiate a conversation with the driver using generative AI model 114. Generative AI model 114 may generate a question for the driver, and cognitive load management system 108 may output the question to a speaker 132 using text to voice converter 116. The driver may respond to the question via microphone 170. By engaging with the cognitive load management system 108 and responding to the question, the cognitive load of the driver may increase above the first cognitive load threshold, reducing the drowsiness of the driver and increasing a safety of the driver and the vehicle.

[0032] FIG. 2 is an information flow diagram 200 showing a flow of information during processing of the cognitive load of the driver by a cognitive load management system such as cognitive load management system 108. An in-vehicle camera 204, such as DMS camera, may capture images of a driver 202. A face and landmark detection task 206 may be performed on the captured images, by the DMS system or by a separate component. Face and landmark data extracted from the captured images may be inputted into a cognitive load estimator 208 (e.g., cognitive load estimator 172), which may estimate a cognitive load of driver 202 based on the face and landmark data. Cognitive load estimator 208 may output a cognitive load score for driver 202, which may be inputted into a cognitive threshold detector 210 (e.g., cognitive threshold detector 110), which may determine whether the cognitive load score is less than a first threshold value, or greater than a second threshold value. If the cognitive load score is less than the first threshold value, the cognitive load management system may select a cognitive engagement strategy 212 for engaging driver 202, to prevent a drowsiness and increase a focus of driver 202. A cognitive stimulator 214 (e.g., cognitive stimulator 112) may execute cognitive engagement strategy 212. In particular, cognitive stimulator 214 may engage driver 202 in conversation, by generating an audio stream at one or more in-vehicle speakers 222, and receiving voice input from driver 202 via an in-vehicle microphone 224. By engaging driver 202 in conversation, the cognitive load of the driver may be increased, thereby increasing a safety of the driver. If the cognitive load score is greater than the first threshold, then the cognitive load management system may adjust one or more cabin controls 220 (e.g., cabin controls 136) to reduce the driver’s cognitive load.

[0033] Referring now to FIG. 3, an example method 300 is shown for monitoring a cognitive load of a driver (e.g., driver 202) of a vehicle, and intervening if the cognitive load increases or decreases to a level at which driver safety may be affected. Method 300 may beAttorney Docket No. P240098WO performed by a cognitive load management system of the vehicle, such as cognitive load management system 108 of FIG.1. Instructions for performing method 300 may be stored in a memory of the vehicle and executed on a processor of the controller, such as memory 106 and processor 104, respectively.

[0034] At 302, method 300 includes estimating and / or measuring vehicle operating conditions. For example, the vehicle operating conditions may include, but are not limited to, a status of an engine of the vehicle (e.g., whether the engine is switched on), and an engagement of one or more gears of a transmission of the vehicle (e.g., whether the vehicle is moving). Vehicle operating conditions may include engine speed and load, vehicle speed, transmission oil temperature, exhaust gas flow rate, mass air flow rate, coolant temperature, coolant flow rate, engine oil pressures (e.g., oil gallery pressures), operating modes of one or more intake valves and / or exhaust valves, electric motor speed, battery charge, engine torque output, vehicle wheel torque, and so on. In one example, the vehicle is a hybrid electric vehicle, and estimating and / or measuring vehicle operating conditions includes determining whether the vehicle is being powered by an engine or an electric motor.

[0035] At 304, method 300 includes determining whether the cognitive load management system has been activated. In some embodiments, the cognitive load management system may be activated upon vehicle startup. In other embodiments, cognitive load management system may be activated by the driver, for example, by selecting a control on the dashboard of the vehicle, such as within a UI of an IVI system (e.g., IVI system 134).

[0036] If at 304 it is determined that the cognitive load management system has not been activated, method 300 proceeds to 306. At 306, method 300 includes waiting until the cognitive load management system has been activated, and method 300 proceeds back to 304. Alternatively, if at 304 it is determined that the cognitive load management system has been activated, method 300 proceeds to 308.

[0037] At 308, method 300 includes determining, when the cognitive load management system is activated, whether it is a first activation of the cognitive load system, meaning that the cognitive load management system has not been previously activated. If the cognitive load management system has been previously activated, method 300 proceeds to 322. At 322, method 300 includes monitoring the cognitive load of the driver and intervening when conditions are met, which is described in reference to FIG.5.

[0038] If at 308 the cognitive load management system has not been previously activated, method 300 proceeds to 310. At 310, method 300 includes generating an interest profile of the driver. The interest profile may include a collection or list of various interests of the driver,Attorney Docket No. P240098WO which may be used by a cognitive stimulator (e.g., cognitive stimulator 214 of FIG. 2) the cognitive load management system when engaging the driver in conversation using a generative AI model (e.g., generative AI model 114). The use of the cognitive stimulator is described in greater detail below in reference to FIG.4.

[0039] At 312, generating the interest profile of the driver includes prompting the driver to opt into the cognitive load management system using a smart phone companion app. In various embodiments, the driver may be prompted to download the companion app via the IVI system of vehicle when the driver first activates the cognitive load management system. The driver may download the companion app and install the companion app on a smart phone of the driver. By installing the companion app, the driver may be opted into the cognitive load management system of the vehicle. In some embodiments, the driver may be requested to manually opt into the cognitive load management system using the smart phone.

[0040] At 314, method 300 includes receiving an opt-in notification from the smart phone of the driver. That is, when the driver opts into the cognitive load management system, the smart phone may send a notification to the cognitive load management system indicating that the driver has opted in. When the cognitive load management system receives the notification, the cognitive load management system may retrieve (e.g., upload) data about the driver from the smart phone of the driver.

[0041] At 316, method 300 includes extracting data from the smart phone of the driver via the companion app about interests of the driver, which may be determined in various ways. In particular, the data may include a list of apps installed on the smart phone. The data may also include settings or preferences of one or more apps included in the list of apps. For example, if the driver has installed a news app on the smart phone, data about news categories that the driver is interested in may be uploaded from the smart phone; if a music app has been installed on the smart phone, information about a type of music that the driver likes may be uploaded from the smart phone; if a sports app has been installed on the smart phone, information about sports that the driver is interested in may be uploaded from the smart phone; and so on. In some embodiments, histories of user interaction with large language models (LLM) using the smart phone, such as questions and prompts created by the driver.

[0042] Additionally, information from messaging, email, productivity and / or communications apps may be extracted, such as emails or text messages of the driver. However, extracting information of this type may present a greater security risk to the driver, where the driver may be more concerned about vehicle systems accessing personal information included in the emails, text messages, etc. In some examples, the opt-in notification mayAttorney Docket No. P240098WO prompt the driver to select a security classification of a plurality of security classifications, where the selected security classification indicates a type of information to be extracted. For example, the driver may select a first security classification, and information from entertainment and music apps on the smart phone may be extracted; the driver may select a second security classification, and information from text messages may be extracted; the driver may select a third security classification, and information from email messages on the smart phone may be extracted; and so on. Thus, an added technical benefit of the method is that the driver can maintain personal data of the driver at a desired level of security, and release selected personal data while withholding other personal data.

[0043] At 318, method 300 includes creating an interest profile for the driver based on the data extracted from the smart phone of the driver via the companion app. In various embodiments, the interest profile may be generated by an interest model (e.g., interest model 118) that takes the extracted data as input, and outputs an encoding of various interests of the driver. In some examples, the encoding may be a vector of values, where each value represents an encoded interest of the driver. For example, a first portion of the vector may include an encoding of sports interests of the driver; a second portion of the vector may include an encoding of musical tastes of the driver; and so on. Some of the encoded portions may include classifications of the driver into a category, based on different criterion in the extracted data. Additionally, in some embodiments, the encoding may include reactions of the driver to stimuli such as changes in cabin lighting, cabin temperature, changes to an HVAC system of the vehicle, and / or reactions to receiving information from the vehicle, including a reaction of the driver to prompts / responses of the generative AI model may be captured by an in-cabin camera, and the reaction may be encoded and stored in the encoding. In this way, the longer the cognitive stimulator is used, the more precise the interventions may be.

[0044] At 320, method 300 includes receiving preferences of the driver with respect to the cognitive load management system from the companion app. That is, the driver may be prompted to enter in data and / or select various options from one or more menus of options. For example, the driver may establish a preference for how the cognitive stimulator initiates a conversation with the driver. The driver may manually enter in topics of interest to discuss, or prioritize topics of interest determined by the cognitive load management system. The driver may set a preference for a type of voice to be used by the cognitive stimulator, such as a male voice or a female voice, a desired accent, etc. The driver may set a preference for when the cognitive load management system is automatically activated or disabled, or a different type ofAttorney Docket No. P240098WO preference. The preferences may be uploaded from the smart phone and stored in the cognitive load management system and / or a memory of the vehicle (e.g., memory 106).

[0045] The interest profile may be stored in the cognitive load management system. After the interest profile has been generated and stored, method 300 proceeds to 322, where method 300 includes monitoring the cognitive load of the driver. Method 300 ends.

[0046] Referring now to FIG.4, an example method 400 is shown for intervening with the driver during operation of the vehicle, when a cognitive load management system determines that a cognitive load of the driver has increased or decreased to a level at which driver safety may be affected. Method 400 may be performed by the cognitive load management system of the vehicle as part of method 300 described above in reference to FIG. 3. Instructions for performing method 400 may be stored in a memory of the vehicle and executed on a processor of the controller, such as memory 106 and processor 104, respectively.

[0047] At 402, method 400 includes determining a driving situation of the driver. Determining the driving situation of the driver may include determining whether the driver is being operated (e.g., whether the vehicle is moving). Determining the driving situation may also include determining whether the driver is operating the vehicle alone or whether there are passengers in the vehicle. A number of passengers in the vehicle may be detected by sensors of vehicle (e.g., vehicle sensors 150), such as weight sensors of passenger seats of the vehicle, or other sensors. Determining the driving situation may also include determining an environment in which the vehicle is being operated. For example, the vehicle may be operating on a highway, or a country road, or an urban environment characterized by stop and go traffic. The vehicle may be operating on a straight route with few turns, or the vehicle may be operating on a winding road with many curves. The environment in which the vehicle is being operated may be determined based on input from a navigational system of the vehicle (e.g., navigational system 154) and / or a type of operation of the vehicle. For example, if one or more vehicle sensors, such as brake sensors, accelerator pedal sensors, steering wheel sensors, etc. indicate changes in a velocity and / or direction of the vehicle, it may be inferred that the vehicle is being operated in an urban or residential environment. Alternatively, if few changes in velocity and / or direction of the vehicle are detected, it may be inferred that the driver is operating on a road or highway. In some examples, details about the type of driving and / or the driving environment may be converted into an encoding, which may be used by the cognitive load management system to determine whether to engage with the driver, as described below.

[0048] At 404, method 400 includes collecting DMS data and in-cabin sensor data of the driver. The DMS data may include images captured of the driver by an in-cabin camera of aAttorney Docket No. P240098WO DMS system (e.g., DMS 171) of the vehicle. The in-cabin sensor data may additionally include biometric data of the driver acquired via a steering wheel of the driver, in some examples. The DMS data and the in-cabin sensor data may be used to determine a psychological state of the driver. In particular, the DMS data and the in-cabin sensor data may be used to determine the cognitive load of the driver.

[0049] At 406, method 400 includes estimating the cognitive load of the driver based on the DMS data and the in-cabin sensor data. The cognitive load may be estimated by a cognitive load estimator (e.g., cognitive load estimator 172) using methods and techniques known in the art, which may be part of the DMS system or maybe a separate component of the vehicle that receives input data from the DMS system, as described above. The cognitive load estimator may output a cognitive load score of the driver. The cognitive load score may indicate or summarize the cognitive load of the driver. The cognitive load score may be a value expressed within a range of values representing different cognitive loads. For example, a cognitive load of 10 may indicate a maximum cognitive load of the driver, where the driver has many cognitive tasks demanding their attention. A cognitive load of zero may indicate a minimum cognitive load of the driver, where the driver has no cognitive tasks demanding their attention. A cognitive load of five may indicate an average cognitive load of the driver, where the driver has a number of cognitive tasks that permits safe driving, where a level of distraction is low, but not so low that the driver becomes drowsy.

[0050] At 408, method 400 includes determining whether the cognitive load is less than a first threshold. The first threshold may be a value at which the low cognitive load of the driver may begin to cause drowsiness in the driver, which may reduce a safety of the driver and the vehicle. For example, the first threshold may be two, where if the cognitive load score of the driver is less than two, then the driver may be at a risk of experiencing drowsiness.

[0051] If at 408 it is determined that the cognitive load score of the driver is less than the second threshold, method 400 proceeds to 410. At 410, method 400 includes engaging the driver in conversation using a generative AI model (e.g., generative AI model 114). Engaging the driver in conversation using the generative AI model may include, at 412, selecting a topic of interest of the driver from an interest profile of the driver, which may be generated as described above in reference to method 300 of FIG. 3. Further, in various embodiments, the topic of interest may be selected based on the cognitive load score of the driver, by consulting a mapping of different topics of interest to different cognitive load scores (e.g., cognitive load mappings 124).Attorney Docket No. P240098WO

[0052] The engagement of the driver is shown in more detail in FIG. 5, which shows an expanded data flow diagram 500 of a flow of data during the engagement of the driver 202, with reference to the components of the cognitive load management system introduced in FIGS. 1 and 2, and in particular, cognitive stimulator 214 of FIG.2. Cognitive stimulator 214 may rely on generative AI model 114 of FIG.1 to engage driver 202 in conversation. Cognitive stimulator 214 may select one or various topics of conversation to engage driver 202, that are of interest to driver 202. To select the one or various topics, cognitive stimulator 214 may receive or retrieve data from interest profile 122, which may indicate topics of interest, entertainment preferences, etc., of the driver. Interest profile 122 may be previously generated by the driver via a companion application (app) 504 of the cognitive load management system, which may be downloaded by the driver and installed on a smart phone 502 of driver 202 at a previous time. Cognitive stimulator 214 may upload data about driver 202 from companion app 504 to generate interest profile 122, as described above in reference to method 300. Cognitive stimulator 214 may then cross reference the retrieved topics of interest with cognitive load mappings 124, to determine an appropriate topic of interest to select, based on the cognitive load score of the driver. In other words, for each retrieved topic of interest, a corresponding cognitive load score associated with the retrieved topic of interest may be retrieved from cognitive load mappings 124. If the cognitive load score of the retrieved topic of interest matches the driver’s cognitive load score, the retrieved topic of interest may be selected for starting the conversation with the driver. In this way, topics of interest that are suitable for the driver’s cognitive load may be selected. That is, a topic of interest may be selected that will increase the driver’s cognitive load to above the first threshold of method 400. Thus, if the driver’s cognitive load is slightly below the first threshold, a first topic of interest may be selected that engages the driver to a first degree (e.g., the weather, sports, or other light conversational topics). If the driver’s cognitive load is further below the first threshold, a second topic of interest may be selected that engages the driver to a second, greater degree, where the driver may exert more cognitive effort (e.g., greater cognitive load) to participate in the conversation. For example, the second topic of interest may be a political topic, a topic from a news feed of the driver, a topic regarding a hobby of driver, etc.

[0053] To initiate a conversation with driver 202, cognitive stimulator 214 may generate a prompt for generative AI model 114. The prompt may include instructions for initiating the conversation. For example, the prompt may include textual instructions such as “start a conversation with a person on the following topic of interest”, where the topic of interest is selected from interest profile 122 based on the cognitive load mappings 124. Further, in someAttorney Docket No. P240098WO embodiments, the prompt may provide more detailed instructions to generative AI model 114 to engage the driver based on the cognitive load score. For example, cognitive stimulator 214 may prompt generative AI model 114 to select a question to ask the driver on the topic of interest that demands an amount of thought in responding. For example, the prompt may instruct generative AI model 114 to ask an easy question, a question that involves a complicated or multi-part response, a demanding question, a question to which there may be various answers, etc.

[0054] Generative AI model 114 may receive the prompt and generate text for initiating the conversation. Cognitive stimulator 214 may convert the text for initiating the conversation into a first audio stream 507 using text-to-voice converter 116. First audio stream 507 may be outputted to a speaker 222 via a wired or wireless network of the vehicle. Driver 202 may hear first audio stream 507 including the question on the speaker 222, and the driver may respond to the question via microphone 224. Microphone 224 may generate a second audio stream 508, which may be transmitted to cognitive stimulator 214 via the wired or wireless network. Second audio stream 508 may be stored in a buffer 506 while driver 202 is responding to the question. Cognitive stimulator 214 may convert the buffered second audio stream 508 in buffer 506 into a textual response using text-to-voice converter 116. The textual response may then be submitted to generative AI model 114 to continue the conversation. Generative AI model 114 may generate a response to the textual response, and the conversation may continue accordingly, until the driver terminates the conversation. In some embodiments, the driver may terminate the conversation using a predefined key phrase, while in other embodiments, generative AI model 114 may process and respond to a variety of natural language statements of the driver indicating a desire to discontinue the conversation.

[0055] In some examples, the conversation may refer to visual content, such as images or video played for driver 202 via IVI system 134. Further, in some examples, the driver may interact with a UI 510 of IVI system 134, and the interaction may be transmitted to cognitive stimulator 214 as part of the response of driver 202. For example, generative AI model 114 may pose a question about an image displayed on IVI system 134, and driver 202 may respond regarding the image. Further, in some embodiments, driver 202 may select a portion of the image via UI 510 with a finger. The selection of the portion of the image may be transmitted to cognitive stimulator 214 in addition to second audio stream 508, or the portion of the image may be transmitted as the response of driver 202 and no second audio stream 508 may be generated.Attorney Docket No. P240098WO

[0056] In other examples, the conversation may not be based on interests of the driver, and the conversation may be based on a “gamification” of elements of the driver’s environment or knowledge. In other words, cognitive stimulator 214 may pose challenge questions to the driver, and generate incentives for correct answers. For example, cognitive stimulator 214 may ask the driver to estimate a distance between the vehicle and a preceding vehicle, or an object in the environment. Cognitive stimulator 214 may ask the driver to identify an object in the environment, or provide information about the object in the environment. A question posed by cognitive stimulator 214 may be based on statistical information collected by the cognitive load management system. For example, cognitive stimulator 214 may record a color of vehicles that the driver has passed on a journey, and ask the driver, “how many blue cars have you passed since the start of your journey”, or ask other types of trivia questions based on data collected by cognitive stimulator 214. If the driver answers a trivia question correctly, an incentive may be provided to the driver, such as, for example, a free-of-charge use of a subscription service of the vehicle available to the driver for a specified duration. For example, if the driver answers a question correctly, the driver may “win” two weeks use of a subscription satellite music service available at the vehicle.

[0057] Returning to FIG. 4, at 414, method 400 includes determining whether conditions are met for terminating the conversation. The conditions for terminating the conversation may include the driver requesting to terminate the conversation, as part of the conversation. If the driver requests to terminate the conversation, the generative AI model may interpret the request, and terminate the conversation. The conditions for terminating the conversation may also include certain changes in the operation of the vehicle by the driver. For example, the conversation may be terminated when a brake sensor of the vehicle (e.g., of the vehicle sensors 150) detects an application of the brake by the driver above a threshold application, or if a steering wheel sensor detects a shift in a direction of the vehicle above a threshold or within a threshold amount of time. In other words, if a sudden change in the operation of the vehicle is detected, it may be inferred that the cognitive load of the driver has increased above the first threshold.

[0058] Additionally, in some embodiments, the conditions for terminating the conversation may be met when the cognitive load estimator detects that the cognitive load of the driver has increased above the first threshold. If the cognitive load estimator detects that the cognitive load of the driver has increased above the first threshold, the cognitive load stimulator may prompt the generative AI model to inform the driver that the driver’s cognitive engagement with the task of driving appears to be sufficiently high that the conversation may no longer beAttorney Docket No. P240098WO helpful to the driver, and ask the driver if the driver wishes to discontinue the conversation. If the driver wishes to discontinue the conversation, the cognitive load stimulator may terminate the conversation. If the driver wishes to continue the conversation, the cognitive load stimulator may continue with the conversation.

[0059] If the conversation is terminated, at 416, method 400 includes collecting feedback about an effectiveness of the cognitive engagement strategy. The feedback may include a duration of the conversation. The feedback may also include ratings of the driver’s responses to the conversation. For example, when the cognitive load of the driver is low, it may be determined that questions about the news raise the driver’s cognitive load, and a first rating of the news questions at increasing the driver’s cognitive load may be stored. The cognitive load stimulator may then try conversing about a different topic, such as sports, and the cognitive load of the driver may be detected as increasing to a greater degree. A second rating of the sports questions at increasing the driver’s cognitive load may be stored. The stored ratings may then be used to optimize a performance of the cognitive load stimulator. Method 400 ends.

[0060] If at 408 it is determined that the cognitive load of the driver is not less than the first threshold, method 400 proceeds to 418. At 418, method 400 includes determining whether the cognitive load score of the driver is above a second threshold value. The second threshold value may be of value at which a safety of the driver may be affected by the cognitive load. For example, the second threshold value may be eight, out of a scale of 10, where if the cognitive load score is greater than eight, the safety of the driver may be affected by the high cognitive load. If at 418 it is determined that the cognitive load score of the driver is greater than the second threshold value, method 400 proceeds to 420.

[0061] At 420, method 400 includes adjusting cabin controls (e.g., cabin controls 136) of the vehicle to decrease the cognitive load of the driver. In some embodiments, adjusting the cabin controls may include adjusting an interior lighting of the cabin. For example, light with a frequency closer to a blue spectrum may induce focus and concentration in the driver to a higher degree than light with a frequency further from the blue spectrum. Therefore, the interior lighting of the cabin may be adjusted to increase the focus and concentration of the driver. Similarly, adjusting the cabin controls to reduce the level of distraction of the driver may also include adjusting a temperature of the cabin. In various embodiments, an adjustment to an interior lighting of the cabin may pertain to an intensity of the lighting, a frequency of the lighting, a pattern of the lighting, and / or a color of the lighting. For various embodiments, instead of or in addition to an adjustment to a temperature of the cabin, an adjustment may beAttorney Docket No. P240098WO made to an HVAC setting, an air flow, an audio signal, an audio alert, and / or visual notifications (e.g., using a DC / IVI system).

[0062] In some embodiments, adjusting the cabin controls to reduce the cognitive load of the driver may include altering one or more settings of an audio system of the vehicle, or prompting the driver to alter the one or more settings of the audio system. Certain types of audio content may be more calming and conducive to increasing the focus of the driver than other types of audio content. For example, the cognitive load management system may adjust the cabin controls to decrease a volume of the music, and / or prompt the driver to change the music to a different type of music.

[0063] In some examples, the cognitive load management system may issue an audio notification alerting the driver to the high cognitive load of the driver. The controller may further provide to the driver one or more suggested adjustments to the cabin controls for selection by the driver. In some embodiments, the controller may suggest a different type of audio content that may be associated with a more energetic mood of the driver. As another example, a fan speed of an HVAC system of the vehicle may be increased, and / or IVI content may be minimized. After adjusting the cabin controls, method 400 may proceed back to 404 to continue collecting the DMS data and monitoring the cognitive load of the driver.

[0064] Alternatively, if at 418 it is determined that the cognitive load is not greater than the second threshold, method 400 proceeds to 422. At 422, method 400 includes continuing operating conditions of the vehicle, and method 400 ends.

[0065] FIG. 6 shows an interior of a cabin 600 of a vehicle 602, in which a driver and / or one or more passengers may be seated. Vehicle 602 of FIG.6 may be a motor vehicle including drive wheels (not shown) and an internal combustion engine 604. Internal combustion engine 604 may include one or more combustion chambers which may receive intake air via an intake passage and exhaust combustion gases via an exhaust passage. Vehicle 602 may be a road automobile, among other types of vehicles. In some examples, vehicle 602 may include a hybrid propulsion system including an energy conversion device operable to absorb energy from vehicle motion and / or the engine and convert the absorbed energy to an energy form suitable for storage by an energy storage device. Vehicle 602 may include a fully electric vehicle, incorporating fuel cells, solar energy capturing elements, and / or other energy storage systems for powering the vehicle.

[0066] Vehicle 602 may include a plurality of vehicle systems, including a braking system for providing braking, an engine system for providing motive power to wheels of the vehicle, a steering system for adjusting a direction of the vehicle, a transmission system for controllingAttorney Docket No. P240098WO a gear selection for the engine, an exhaust system for processing exhaust gases, and the like. Further, the vehicle 602 includes an in-vehicle computing system 609, which may include elements of vehicle control system 100 of FIG. 1. The in-vehicle computing system 609 is described in greater detail below in reference to FIG.7. The in-vehicle computing system 609 may include a cognitive load management system such as cognitive load management system 108, which may monitor a cognitive load of a driver of vehicle 602, as described above.

[0067] As shown, an instrument panel 606 may include various displays and controls accessible to a human user (e.g., a driver or a passenger) of vehicle 602. For example, instrument panel 606 may include a touch screen 608 of an in-vehicle computing system or infotainment system 609 (e.g., IVI system 134), an audio system control panel, and an instrument cluster 610. Touch screen 608 may receive user input to the in-vehicle computing system or infotainment system 609 for controlling audio output, visual display output, user preferences, control parameter selection, and so on.

[0068] While the example system shown in FIG.6 includes audio system controls that may be performed via a user interface of in-vehicle computing system or infotainment system 609, such as touch screen 608 without a separate audio system control panel, in other embodiments, the vehicle may include an audio system control panel, which may include controls for a conventional vehicle audio system such as a radio, compact disc player, MP3 player, and so on. The audio system controls may include features for controlling one or more aspects of audio output via one or more speakers 612 (e.g., speakers 132) of a vehicle speaker system. For example, the in-vehicle computing system or the audio system controls may control a volume of audio output, a distribution of sound among the individual speakers of the vehicle speaker system, an equalization of audio signals, and / or any other aspect of the audio output. In further examples, in-vehicle computing system or infotainment system 609 may adjust a radio station selection, a playlist selection, a source of audio input (e.g., from radio or CD or MP3), and so on, based on user input received directly via touch screen 608, or based on data regarding the user (such as a cognitive state or load of the driver) received via one or more external devices 650 and / or a mobile device 628. The audio system of the vehicle may include an amplifier (not shown) coupled to plurality of loudspeakers (not shown). In some embodiments, one or more hardware elements of in-vehicle computing system or infotainment system 609, such as touch screen 608, a display screen 611, various control dials, knobs and buttons, memory, processor(s), and any interface elements (e.g., connectors or ports) may form an integrated head unit that is installed in instrument panel 606 of the vehicle. The head unit may be fixedly or removably attached in instrument panel 606. In additional or alternative embodiments, one orAttorney Docket No. P240098WO more hardware elements of the in-vehicle computing system or infotainment system 609 may be modular and may be installed in multiple locations of the vehicle.

[0069] The cabin 600 may include one or more sensors for monitoring the vehicle, the user, and / or the environment. For example, the cabin 600 may include one or more seat-mounted pressure sensors configured to measure the pressure applied to the seat to determine the presence of a user, door sensors configured to monitor door activity, humidity sensors to measure the humidity content of the cabin, microphones to receive user input in the form of voice commands, to enable a user to conduct telephone calls, and / or to measure ambient noise in the cabin 600, and so on. It is to be understood that the above-described sensors and / or one or more additional or alternative sensors may be positioned in any suitable location of the vehicle. For example, sensors may be positioned in an engine compartment, on an external surface of the vehicle, and / or in other suitable locations for providing information regarding the operation of the vehicle, ambient conditions of the vehicle, a user of the vehicle, and so on. Information regarding ambient conditions of the vehicle, vehicle status, or vehicle driver may also be received from sensors external to / separate from the vehicle (that is, not part of the vehicle system), such as sensors coupled to external devices 650 and / or mobile device 628. Sensor data of various sensors of the vehicle may be transmitted to and / or accessed by the in- vehicle computing system 609 via a bus of the vehicle, such as a CAN bus.

[0070] Cabin 600 may also include one or more user objects, such as mobile device 628, that are stored in the vehicle before, during, and / or after travelling. The mobile device 628 may include a smart phone, a tablet, a laptop computer, a portable media player, and / or any suitable mobile computing device. The mobile device 628 may be connected to the in-vehicle computing system via a communication link 630. The communication link 630 may be wired (e.g., via Universal Serial Bus (USB), Mobile High-Definition Link (MHL), High-Definition Multimedia Interface (HDMI), Ethernet, and so on) or wireless (e.g., via Bluetooth®, Wi-Fi®, Wi-Fi Direct®, Near-Field Communication (NFC), cellular connectivity, and so on) and configured to provide two-way communication between the mobile device and the in-vehicle computing system. (Wi-Fi® and Wi-Fi Direct® are registered trademarks of Wi-Fi Alliance, Austin, Texas.) The mobile device 628 may include one or more wireless communication interfaces for connecting to one or more communication links (e.g., one or more of the example communication links described above). The wireless communication interface may include one or more physical devices, such as antenna(s) or port(s) coupled to data lines for carrying transmitted or received data, as well as one or more modules / drivers for operating the physical devices in accordance with other devices in the mobile device. For example, theAttorney Docket No. P240098WO communication link 630 may provide sensor and / or control signals from various vehicle systems (such as vehicle audio system, climate control system, and so on) and the touch screen 608 to the mobile device 628 and may provide control and / or display signals from the mobile device 628 to the in-vehicle systems and the touch screen 608. The communication link 630 may also provide power to the mobile device 628 from an in-vehicle power source in order to charge an internal battery of the mobile device.

[0071] In-vehicle computing system or infotainment system 609 may also be communicatively coupled to additional devices operated and / or accessed by the user but located external to vehicle 602, such as one or more external devices 650. In the depicted embodiment, external devices are located outside of vehicle 602 though it will be appreciated that in alternate embodiments, external devices may be located inside cabin 600. The external devices may include a server computing system, personal computing system, portable electronic device, electronic wrist band, electronic head band, portable music player, electronic activity tracking device, pedometer, smart-watch, GPS system, and so on. External devices 650 may be connected to the in-vehicle computing system via a communication link 636 which may be wired or wireless, as discussed with reference to communication link 630, and configured to provide two-way communication between the external devices and the in-vehicle computing system. For example, external devices 650 may include one or more sensors and communication link 636 may transmit sensor output from external devices 650 to in-vehicle computing system or infotainment system 609 and touch screen 608. External devices 650 may also store and / or receive information regarding contextual data, user behavior / preferences, operating rules, and so on and may transmit such information from the external devices 650 to in-vehicle computing system or infotainment system 609 and touch screen 608.

[0072] In-vehicle computing system or infotainment system 609 may analyze the input received from external devices 650, mobile device 628, and / or other input sources and select settings for various in-vehicle systems (such as climate control system or audio system), provide output via touch screen 608 and / or speakers 612, communicate with mobile device 628 and / or external devices 650, and / or perform other actions based on the assessment. In some embodiments, all or a portion of the assessment may be performed by the mobile device 628 and / or the external devices 650. In particular, the communication link 630 may be used to receive or retrieve, from the mobile device 628, information of the driver stored on the mobile device 628. For example, preferences of the driver, and / or information about interests of the driver may be transmitted from a companion app of the cognitive load management systemAttorney Docket No. P240098WO installed on the mobile device 628 to the cognitive load management system installed in the in- vehicle computing system 609, as described above in reference to FIG.5.

[0073] In some embodiments, one or more of the external devices 650 may be communicatively coupled to in-vehicle computing system or infotainment system 609 indirectly, via mobile device 628 and / or another of the external devices 650. For example, communication link 636 may communicatively couple external devices 650 to mobile device 628 such that output from external devices 650 is relayed to mobile device 628. Data received from external devices 650 may then be aggregated at mobile device 628 with data collected by mobile device 628, the aggregated data then transmitted to in-vehicle computing system or infotainment system 609 and touch screen 608 via communication link 630. Similar data aggregation may occur at a server system and then transmitted to in-vehicle computing system or infotainment system 609 and touch screen 608 via communication link 636 and / or communication link 630.

[0074] FIG. 7 shows a block diagram of an in-vehicle computing system or infotainment system 609 configured and / or integrated inside vehicle 602. In-vehicle computing system or infotainment system 609 may perform one or more of the methods described herein in some embodiments. In some examples, the in-vehicle computing system or infotainment system 609 may be a vehicle infotainment system configured to provide information-based media content (audio and / or visual media content, including entertainment content, navigational services, automated conversational services, and so on) to a vehicle user to enhance the operator’s in- vehicle experience. The in-vehicle computing system or infotainment system 609 may include, or be coupled to, various vehicle systems, sub-systems, hardware components, as well as software applications and systems that are integrated in, or integratable into, vehicle 602 in order to enhance an in-vehicle experience for a driver and / or a passenger.

[0075] In-vehicle computing system or infotainment system 609 may include one or more processors including an operating system processor 714 (e.g., processor 104) and an interface processor 720. Operating system processor 714 may execute an operating system on the in- vehicle computing system, and control input / output, display, playback, and other operations of the in-vehicle computing system. Interface processor 720 may interface with a vehicle control system 730 via an inter-vehicle system communication module 722.

[0076] Inter-vehicle system communication module 722 may output data to one or more other vehicle systems 731 and / or one or more other vehicle control elements 761, while also receiving data input from other vehicle systems 731 and other vehicle control elements 761, e.g., by way of vehicle control system 730. When outputting data, inter-vehicle systemAttorney Docket No. P240098WO communication module 722 may provide a signal via a bus corresponding to any status of the vehicle, the vehicle surroundings, or the output of any other information source connected to the vehicle. Vehicle data outputs may include, for example, analog signals (such as current velocity), digital signals provided by individual information sources (such as clocks, thermometers, location sensors such as GPS sensors, and so on), digital signals propagated through vehicle data networks (such as an engine CAN bus through which engine related information may be communicated, a climate control CAN bus through which climate control related information may be communicated, and a multimedia data network through which multimedia data is communicated between multimedia components in the vehicle). For example, vehicle data outputs may be output to vehicle control system 730, and vehicle control system 730 may adjust vehicle controls 761 based on the vehicle data outputs. As another example, the in-vehicle computing system or infotainment system 609 may retrieve from the engine CAN bus the current speed of the vehicle estimated by the wheel sensors, a power state of the vehicle via a battery and / or power distribution system of the vehicle, an ignition state of the vehicle, and so on. In addition, other interfacing means such as Ethernet may be used as well without departing from the scope of this disclosure.

[0077] A storage device 708 may be included in in-vehicle computing system or infotainment system 609 to store data such as instructions executable by operating system processor 714 and / or interface processor 720 in non-volatile form. The storage device 708 may store application data, including prerecorded sounds, to enable the in-vehicle computing system or infotainment system 609 to run an application for connecting to a cloud-based server and / or collecting information for transmission to the cloud-based server. The application may retrieve information gathered by vehicle systems / sensors, input devices (e.g., a user interface 718), data stored in one or more storage devices, such as a volatile memory 719A or a non-volatile memory 719B, devices in communication with the in-vehicle computing system (e.g., a mobile device connected via a Bluetooth® link), and so on. In-vehicle computing system or infotainment system 609 may further include a volatile memory 719A. Volatile memory 719A may be RAM. Non-transitory storage devices, such as non-volatile storage device 708 (e.g., memory 106) and / or non-volatile memory 719B, may store instructions and / or code that, when executed by a processor (e.g., operating system processor 714 and / or interface processor 720), controls the in-vehicle computing system or infotainment system 609 to perform one or more of the actions described in the disclosure.

[0078] A microphone 702 (e.g., microphone 170) may be included in the in-vehicle computing system or infotainment system 609 to receive voice commands from a user, toAttorney Docket No. P240098WO measure ambient noise in the vehicle, to determine whether audio from speakers of the vehicle is tuned in accordance with an acoustic environment of the vehicle, and so on. In particular, microphone 702 may be used by the driver to engage in a conversation with a cognitive load management system 723 (e.g., cognitive load management system 108), as described herein. A speech processing unit 704 may process voice commands, such as the voice commands received from the microphone 702. The speech processing unit 704 may include a text-to-voice converter, such as text-to-voice converter 116, which may convert text generated by the cognitive load management system 723 into speech that is played at an audio system 732 of the vehicle 602. In some embodiments, in-vehicle computing system or infotainment system 609 may also be able to receive voice commands and sample ambient vehicle noise using a microphone included in the audio system 732.

[0079] One or more additional sensors may be included in a sensor subsystem 710 of the in-vehicle computing system or infotainment system 609. For example, the sensor subsystem 710 may include a plurality of cameras 725, such as a rear view camera for assisting a user in parking the vehicle and / or other external cameras, radars, lidars, ultrasonic sensors, and the like. The sensor subsystem 710 may include an in-cabin camera (e.g., a dashboard cam) for identifying a user (e.g., using facial recognition and / or user gestures). For example, an in-cabin camera may be used to identify one or more users of the vehicle via facial recognition software, and / or to detect a status or state of the one or more users (e.g., a cognitive load). Sensor subsystem 710 of in-vehicle computing system or infotainment system 609 may communicate with and receive inputs from various vehicle sensors and may further receive user inputs. For example, the inputs received by sensor subsystem 710 may include transmission gear position, transmission clutch position, gas pedal input, brake input, transmission selector position, vehicle speed, engine speed, mass airflow through the engine, ambient temperature, intake air temperature, and so on, as well as inputs from climate control system sensors (such as heat transfer fluid temperature, antifreeze temperature, fan speed, passenger compartment temperature, desired passenger compartment temperature, ambient humidity, and so on), an audio sensor detecting voice commands issued by a user, a fob sensor receiving commands from and optionally tracking the geographic location / proximity of a fob of the vehicle, and so on.

[0080] One or more additional sensors may be included in and / or communicatively coupled to a sensor subsystem 710 of the in-vehicle computing system 609. For example, the sensor subsystem 710 may include and / or be communicatively coupled to a camera, such as a rear view camera for assisting a user in parking the vehicle, a cabin camera for identifying aAttorney Docket No. P240098WO user, and / or a front view camera to assess quality of the route segment ahead. The above- described cameras may also be used to provide images to a computer vision-based facial recognition and / or facial analysis module. For example, the facial analysis module may be used to determine an emotional or psychological state of users of the vehicle. Sensor subsystem 710 of in-vehicle computing system 609 may communicate with and receive inputs from various vehicle sensors and may further receive user inputs.

[0081] While certain vehicle system sensors may communicate with sensor subsystem 710 alone, other sensors may communicate with both sensor subsystem 710 and vehicle control system 730, or may communicate with sensor subsystem 710 indirectly via vehicle control system 730. Sensor subsystem 710 may serve as an interface (e.g., a hardware interface) and / or processing unit for receiving and / or processing received signals from one or more of the sensors described in the disclosure.

[0082] A navigation subsystem 711 (e.g., navigational system 154) of in-vehicle computing system or infotainment system 609 may generate and / or receive navigation information such as location information (e.g., via a GPS sensor and / or other sensors from sensor subsystem 710), route information, destination information, traffic information, point- of-interest (POI) identification, and / or provide other navigational services for the user. Navigation sub-system 711 may include inputs / outputs including analog to digital converters, digital inputs, digital outputs, network outputs, radio frequency transmitting devices, and so on. In some examples, navigation sub-system 711 may interface with vehicle control system 730.

[0083] An external device interface 712 of in-vehicle computing system or infotainment system 609 may be coupleable to and / or communicate with one or more external devices 650 located external to vehicle 602. While the external devices are illustrated as being located external to vehicle 602, it is to be understood that they may be temporarily housed in vehicle 602, such as when the user is operating the external devices while operating vehicle 602. In other words, the external devices 650 are not integral to vehicle 602. The external devices 650 may include a mobile device 628 (e.g., connected via a Bluetooth®, NFC, Wi-Fi Direct®, or other wireless connection) or an alternate Bluetooth®-enabled device 752.

[0084] Mobile device 628 may be a mobile phone, smart phone, wearable devices / sensors that may communicate with the in-vehicle computing system via wired and / or wireless communication, or other portable electronic device(s). Other external devices include one or more external services 746. For example, the external devices may include extra-vehicular devices that are separate from and located externally to the vehicle. Still other external devicesAttorney Docket No. P240098WO include one or more external storage devices 754, such as solid-state drives, pen drives, USB drives, and so on. External devices 650 may communicate with in-vehicle computing system or infotainment system 609 either wirelessly or via connectors without departing from the scope of this disclosure. For example, external devices 650 may communicate with in-vehicle computing system or infotainment system 609 through the external device interface 712 over a network 760, a USB connection, a direct wired connection, a direct wireless connection, and / or other communication link.

[0085] One or more applications 744 may be operable on mobile device 628. As an example, a cognitive load management system companion application 744 (e.g., companion app 504) may be operated to aggregate user data regarding interests, applications, preferences, or other data of the user with the mobile device. For example, companion application 744 may aggregate data regarding music playlists listened to by the user on the mobile device, applications downloaded by the user, news channels subscribed to by the user, and so on. The collected data may be transferred by companion application 744 to external device interface 712 over network 760. In addition, specific user data requests may be received at mobile device 628 from in-vehicle computing system or infotainment system 609 via the external device interface 712. The specific data requests may include requests for determining preferences of the user with respect to the cognitive load management system, in one example. Companion application 744 may send control instructions to components of mobile device 628 to enable the requested data to be collected on the mobile device. Companion application 744 may then relay the collected information back to in-vehicle computing system or infotainment system 609.

[0086] Likewise, one or more applications 748 may be operable on external services 746. As an example, external services applications 748 may be operated to aggregate and / or analyze data from multiple data sources. For example, external services applications 748 may aggregate data from one or more social media accounts of the user, data from the in-vehicle computing system (e.g., sensor data, log files, user input, and so on), data from an internet query (e.g., weather data, POI data), and so on. The collected data may be transmitted to another device and / or analyzed by the application to determine a context of the driver, vehicle, and environment and perform an action based on the context (e.g., requesting / sending data to other devices).

[0087] Vehicle control system 730 may include controls for controlling aspects of various vehicle systems 731 involved in different in-vehicle functions. These may include, for example, controlling aspects of vehicle audio system 732 for providing audio entertainment to the vehicleAttorney Docket No. P240098WO occupants, aspects of a climate control system 734 for meeting the cabin cooling or heating needs of the vehicle occupants, as well as aspects of a telecommunication system 736 for enabling vehicle occupants to establish telecommunication linkage with others.

[0088] Audio system 732 may include one or more acoustic reproduction devices including electromagnetic transducers such as one or more speakers 735. Vehicle audio system 732 may be passive or active such as by including a power amplifier. In some examples, in-vehicle computing system or infotainment system 609 may be a sole audio source for the acoustic reproduction device or there may be other audio sources that are connected to the audio reproduction system (e.g., external devices such as a mobile phone). The connection of any such external devices to the audio reproduction device may be analog, digital, or any combination of analog and digital technologies.

[0089] Climate control system 734 may be configured to provide a comfortable environment within the cabin or passenger compartment of vehicle 602. Climate control system 734 includes components enabling controlled ventilation such as air vents, a heater, an air conditioner, an integrated heater and air-conditioner system, and so on. Other components linked to the heating and air-conditioning setup may include a windshield defrosting and defogging system capable of clearing the windshield and a ventilation-air filter for cleaning outside air that enters the passenger compartment through a fresh-air inlet.

[0090] Vehicle control system 730 may also include controls for adjusting the settings of various vehicle control elements 761 (or vehicle controls, or vehicle system control elements) related to the engine and / or auxiliary elements within a cabin of the vehicle, such as one or more steering wheel controls 762 (e.g., steering wheel-mounted audio system controls, cruise controls, windshield wiper controls, headlight controls, turn signal controls, and so on), instrument panel controls, microphone(s), accelerator / brake / clutch pedals, a gear shift, door / window controls positioned in a driver or passenger door, seat controls, cabin light controls, audio system controls, cabin temperature controls, and so on. Vehicle control elements 761 may also include internal engine and vehicle operation controls (e.g., engine controller module, actuators, valves, and so on) that are configured to receive instructions via the CAN bus of the vehicle to change operation of one or more of the engine, exhaust system, transmission, and / or other vehicle system. The control signals may also control audio output at one or more speakers 735 of the vehicle’s audio system 732. For example, the control signals may adjust audio output characteristics such as volume, equalization, audio image (e.g., the configuration of the audio signals to produce audio output that appears to a user to originate from one or more defined locations), audio distribution among a plurality of speakers, and soAttorney Docket No. P240098WO on. Likewise, the control signals may control vents, air conditioner, and / or heater of climate control system 734. For example, the control signals may increase delivery of cooled air to a specific section of the cabin. For example, the control signals may increase delivery of cooled air to a specific section of the cabin.

[0091] Vehicle controls 761 may include a steering control system 762, a braking control system 763, and an acceleration control system 764. Vehicle controls 761 may include additional control systems. In some example, vehicle controls 761 may be operated autonomously. In other examples, vehicle controls 761 may be controlled by a user. Further, in some examples, a user may primarily control vehicle controls 761, while one or more ADAS 765 may intermittently adjust vehicle controls 761 in order to increase vehicle performance. For example, the one or more ADAS 765 may include a cruise control system, a lane departure warning system, a collision avoidance system, an adaptive braking system, and the like.

[0092] Steering control system 762 may be configured to control a direction of the vehicle. For example, during a non-autonomous mode of operation, steering control system 762 may be controlled by a steering wheel. For example, the user may turn the steering wheel in order to adjust a vehicle direction. During an autonomous mode of operation, steering control system 762 may be controlled by vehicle control system 730. In some examples, one or more ADAS 765 may adjust steering control system 762. For example, the vehicle control system 730 may determine that a change in vehicle direction is requested, and may change the vehicle direction via controlling the steering control system 762. For example, vehicle control system 730 may adjust axles of the vehicle in order to change the vehicle direction.

[0093] Braking control system 763 may be configured to control an amount of braking force applied to the vehicle. For example, during a non-autonomous mode of operation, braking control system 763 may be controlled by a brake pedal. For example, the user may depress the brake pedal in order to increase an amount of braking applied to the vehicle. During an autonomous mode of operation, braking system 763 may be controlled autonomously. For example, the vehicle control system 730 may determine that additional braking is requested, and may apply additional braking. In some examples, the autonomous vehicle control system may depress the brake pedal in order to apply braking (e.g., to decrease vehicle speed and / or bring the vehicle to a stop). In some examples, the one or more ADAS 765 may adjust braking control system 763.

[0094] Acceleration control system 764 may be configured to control an amount of acceleration applied to the vehicle. For example, during a non-autonomous mode of operation, acceleration control system 764 may be controlled by an acceleration pedal. For example, theAttorney Docket No. P240098WO user may depress the acceleration pedal in order to increase an amount of torque applied to wheels of the vehicle, causing the vehicle to accelerate in speed. During an autonomous mode of operation, acceleration control system 764 may be controlled by vehicle control system 730. In some examples, the one or more ADAS 765 may adjust acceleration control system 764. For example, vehicle control system 730 may determine that additional vehicle speed is requested, and may increase vehicle speed via acceleration. In some examples, vehicle control system 730 may depress the acceleration pedal in order to accelerate the vehicle. As an example of an ADAS 765 adjusting acceleration control system 764, the ADAS 765 may be a cruise control system, and may include adjusting vehicle acceleration in order to maintain a desired speed during vehicle operation.

[0095] Control elements positioned on an outside of a vehicle (e.g., controls for a security system) may also be connected to in-vehicle computing system or infotainment system 609, such as via inter-vehicle system communication module 722. The control elements of the vehicle control system may be physically and permanently positioned on and / or in the vehicle for receiving user input. In addition to receiving control instructions from in-vehicle computing system or infotainment system 609, vehicle control system 730 may also receive input from one or more external devices 650 operated by the user, such as from mobile device 628. This allows aspects of vehicle systems 731 and vehicle control elements 761 to be controlled based on user input received from the external devices 650.

[0096] In-vehicle computing system or infotainment system 609 may further include one or more antennas 706. The in-vehicle computing system may obtain broadband wireless internet access via antennas 706, and may further receive broadcast signals such as radio, television, weather, traffic, and the like. The in-vehicle computing system or infotainment system 609 may receive positioning signals such as GPS signals via antennas 706. The in- vehicle computing system may also receive wireless commands via radio frequency (RF) such as via antennas 706 or via infrared or other means through appropriate receiving devices. In some embodiments, antenna 706 may be included as part of audio system 732 or telecommunication system 736. Additionally, antenna 706 may provide AM / FM radio signals to external devices 650 (such as to mobile device 628) via external device interface 712.

[0097] One or more elements of the in-vehicle computing system or infotainment system 609 may be controlled by a user via user interface 718 (e.g., UI 510). User interface 718 may include a graphical user interface presented on a touch screen, such as touch screen 608 and / or display screen 611 of FIG.6, and / or user-actuated buttons, switches, knobs, dials, sliders, and so on. For example, user-actuated elements may include steering wheel controls, door and / orAttorney Docket No. P240098WO window controls, instrument panel controls, audio system settings, climate control system settings, and the like. A user may also interact with one or more applications of the in-vehicle computing system or infotainment system 609 and mobile device 628 via user interface 718, or the cognitive load management system 723. In addition to receiving a user’s vehicle setting preferences on user interface 718, vehicle settings selected by in-vehicle control system may be displayed to a user on user interface 718. Notifications and other messages (e.g., received messages), as well as navigational assistance, may be displayed to the user on a display of the user interface. User preferences / information and / or responses to presented messages may be performed via user input to the user interface.

[0098] The in-vehicle computing system or infotainment system 609 may include a DMS 721. The DMS 721 may receive data from various sensors and / or systems of the vehicle (e.g., sensor subsystem 710, cameras 725, microphone 702) and may monitor aspects of driver behavior to improve a performance of the vehicle and / or a driving experience of the driver. For example, an output of the DMS 721 may be used to adjust a cabin lighting of the vehicle, or a temperature of the vehicle, if the DMS 721 detects driver behavior indicative of high cognitive load. In some examples, one or more outputs of the DMS 721 may be inputs into the cognitive load management system 723. In various embodiments, the cognitive load management system 723 may be used to predict a cognitive load of the driver, and adjust one or more controls of the vehicle control system 730 based on the predicted cognitive load of the driver.

[0099] Thus, a cognitive load management system is proposed, which may continuously monitor a cognitive load of the driver to determine whether the cognitive load is outside a range deemed safe for operating the vehicle. A hybrid approach is taken by the cognitive load management system to intervening when a detected or estimated cognitive load of the driver either decreases below a first threshold cognitive load, which may indicate a high level of drowsiness of the driver, or increases above a second threshold cognitive load, which may indicate a high level of stress of the driver. When the detected or estimated cognitive load of the driver decreases below the first threshold cognitive load, where drowsiness or disengagement may impair a performance of the driver, the cognitive load management system may adopt a first strategy to increase the cognitive load of the driver, where a cognitive stimulator component of the system may engage the driver in conversation using a generative AI model. When the detected or estimated cognitive load of the driver increases above the second threshold cognitive load, the cognitive load management system may adopt a second strategy to reduce the cognitive load of the driver. For example, the second strategy may include adjusting one or more controls of the vehicle to reduce a stress of the driver. In thisAttorney Docket No. P240098WO way, high cognitive loads may be decreased and low cognitive loads may be increased by different strategies that accomplish different goals.

[0100] In particular, when the cognitive load of the driver decreases below the first threshold, a cognitive engagement strategy may be determined for engaging the driver, based on information of the driver extracted from a smart phone of the driver via a companion application of the cognitive load management system. A cognitive stimulator component of the cognitive load management system may initiate a conversation based on the cognitive engagement strategy. The cognitive engagement strategy may include selecting topics of conversation based on an interest profile of the driver generated from the information extracted from the smart phone. For example, the cognitive stimulator may ask the driver questions, via an audio system of the vehicle, and the driver may respond via a microphone of the vehicle. Additionally or alternatively, the cognitive stimulator may ask the driver questions about an environment of the vehicle, to stimulate the driver’s attention to the environment. By selecting engaging topics of conversation in this manner, the cognitive stimulator may maintain a conversation with the driver until the driver’s cognitive load has increased above the first threshold or the driver has reached their destination, thereby preventing the driver from becoming drowsy, and increasing the safety of the driver.

[0101] The technical effect of engaging the driver in topics of conversation based on an interest profile of the driver while the driver is operating the vehicle with a cognitive load that is less than a threshold cognitive load, where the interest profile is generated by extracting information from a smart phone of the driver, is that the topics of conversation that are generated may be more engaging to the driver, and may increase their cognitive load faster and more efficiently than randomly selected topics of conversation. By engaging the driver more quickly, a time of the driver spent using the cognitive stimulator may be reduced, leading to an overall reduction in processing and computation performed by the cognitive stimulator.

[0102] The disclosure also provides support for a cognitive load management system of a vehicle, comprising: a processor, and a memory storing instructions that when executed, cause the processor to: during operation of the vehicle by a driver: estimate a cognitive load of the driver based on information received from a driver monitoring system (DMS) of the vehicle, generate a cognitive load score of the driver based on the estimated cognitive load, detect that the cognitive load score is below a first threshold value, and in response, engage the driver in conversation using a generative artificial intelligence (AI) model to increase the cognitive load. In a first example of the system, further instructions are stored in the memory that when executed, cause the processor to: detect that the cognitive load score is above a second thresholdAttorney Docket No. P240098WO value, and in response, adjust one or more cabin controls of the vehicle to reduce the cognitive load. In a second example of the system, optionally including the first example, the information received from the DMS includes at least one of: output of a dashboard camera of the vehicle, seat occupancy data of the vehicle, and biometric data of the driver. In a third example of the system, optionally including one or both of the first and second examples, further instructions are stored in the memory that when executed, cause the processor to: retrieve an interest profile of the driver from the memory, the interest profile including topics that the driver is interested in, submit a prompt to the generative AI model to initiate the conversation by asking a question to the driver based on the topics, convert the question outputted by the generative AI model into an audio stream using a text-to-voice converter of the cognitive load management system, and output the audio stream to a speaker of the vehicle. In a fourth example of the system, optionally including one or more or each of the first through third examples, further instructions are stored in the memory that when executed, cause the processor to: select a topic of the interest profile based on the cognitive load score of the driver, according to a cognitive load mapping stored in the memory, the cognitive load mapping indicating topics of the interest profile that match the cognitive load score. In a fifth example of the system, optionally including one or more or each of the first through fourth examples, the interest profile is generated from information about the driver extracted from a smart phone of the driver by a companion application of the cognitive load management system, the companion application installed on the smart phone. In a sixth example of the system, optionally including one or more or each of the first through fifth examples, the information about the driver extracted from the smart phone is based on one or more of: one or more apps installed on the smart phone, preferences of an app installed on the smart phone, emails and / or text messages stored on the smart phone, histories of interactions with large language models (LLMs) using the smart phone, and preferences of the driver with respect to the cognitive load management system included in the companion application. In a seventh example of the system, optionally including one or more or each of the first through sixth examples, the information about the driver extracted from the smart phone is used to classify the driver into a category, and the topic is selected based on the category. In a eighth example of the system, optionally including one or more or each of the first through seventh examples, further instructions are stored in the memory that when executed, cause the processor to: receive an audio stream from the driver via a microphone of the vehicle, the audio stream including a response to the question, convert the received audio stream into text using the text-to-voice converter, enter the text into the generative AI model to continue the conversation. In a ninth example of the system, optionallyAttorney Docket No. P240098WO including one or more or each of the first through eighth examples, further instructions are stored in the memory that when executed, cause the processor to: prompt the generative AI model to gamify the conversation by providing challenge questions, and offering incentives as rewards for correct answers. In a tenth example of the system, optionally including one or more or each of the first through ninth examples, the rewards include a free-of-charge use of a subscription service of the vehicle available to the driver for a specified duration. In a eleventh example of the system, optionally including one or more or each of the first through tenth examples, the challenge questions include questions based on statistical information collected by the cognitive load management system during a journey of the vehicle.

[0103] The disclosure also provides support for a method for a cognitive load management system of a vehicle, the method comprising: collecting in-cabin sensor information of a driver of the vehicle during operation of the vehicle, the in-cabin sensor information including data collected from a driver monitoring system (DMS) of the vehicle, estimating a cognitive load of the driver based on the collected in-cabin sensor information, generating a cognitive load score of the driver based on the estimated cognitive load, detecting that the cognitive load score is below a first threshold value, and in response, engaging the driver in conversation to increase the cognitive load using a generative artificial intelligence (AI) model. In a first example of the method, the in-cabin sensor information includes at least one of: an output of a dashboard camera of the vehicle, seat occupancy data of the vehicle, and biometric data of the driver. In a second example of the method, optionally including the first example, engaging the driver in conversation to increase the cognitive load using the generative AI model further comprises: retrieving an interest profile of the driver from a memory of the cognitive load management system, the interest profile including topics that the driver is interested in, submitting a prompt to the generative AI model to initiate the conversation by asking a question to the driver based on the topics, converting the question outputted by the generative AI model into an audio stream using a text-to-voice converter, outputting the audio stream to a speaker of the vehicle, receiving an audio stream from the driver via a microphone of the vehicle, the audio stream including a response to the question, converting the received audio stream into text using the text-to-voice converter, and entering the text into the generative AI model. In a third example of the method, optionally including one or both of the first and second examples, submitting the prompt to the generative AI model to initiate the conversation further comprises: selecting a topic of the interest profile based on at least one of: a category that the driver is classified into by the cognitive load management system, based on the interest profile, the cognitive load score of the driver, according to a cognitive load mapping indicating topics of the interestAttorney Docket No. P240098WO profile that match the cognitive load score. In a fourth example of the method, optionally including one or more or each of the first through third examples, the interest profile is generated from information about the driver extracted from a smart phone of the driver by a companion application of the cognitive load management system, the companion application installed on the smart phone. In a fifth example of the method, optionally including one or more or each of the first through fourth examples, the method further comprises: prompting the generative AI model to gamify the conversation by providing challenge questions and offering incentives as rewards for correct answers, the challenge questions including questions based on statistical information collected by the cognitive load management system during a journey of the vehicle, the rewards including a free-of-charge use of a subscription service of the vehicle available to the driver for a specified duration.

[0104] The disclosure also provides support for a method for a cognitive load management system of a vehicle, the method comprising: wirelessly receiving information from a companion application of the cognitive load management system installed on a smart phone of a driver of the vehicle, the information including one or more of the following: one or more apps installed on the smart phone, preferences of an app installed on the smart phone, emails and / or text messages stored on the smart phone, and histories of interactions with large language models (LLMs) using the smart phone, creating an interest profile of the driver based on the received wireless information, the interest profile including topics of interest of the driver, and engaging the driver in conversation based on a topic of interest of the interest profile, using a generative artificial intelligence (AI) model. In a first example of the method, the topic of interest is selected by the cognitive load management system based on an estimated cognitive load of the driver.

[0105] The description of embodiments has been presented for purposes of illustration and description. Suitable modifications and variations to the embodiments may be performed in light of the above description or may be acquired from practicing the methods. For example, unless otherwise noted, one or more of the described methods may be performed by a suitable device and / or combination of devices, such as the embodiments described above with respect to FIGS.1-7. The methods may be performed by executing stored instructions with one or more logic devices (e.g., processors) in combination with one or more hardware elements, such as storage devices, memory, hardware network interfaces / antennas, switches, clock circuits, and so on. The described methods and associated actions may also be performed in various orders in addition to the order described in this application, in parallel, and / or simultaneously. The described systems are exemplary in nature, and may include additional elements and / or omitAttorney Docket No. P240098WO elements. The subject matter of the present disclosure includes all novel and non-obvious combinations and sub-combinations of the various systems and configurations, and other features, functions, and / or properties disclosed.

[0106] As used in this application, an element or step recited in the singular and proceeded with the word “a” or “an” should be understood as not excluding plural of said elements or steps, unless such exclusion is stated. Furthermore, references to “one embodiment” or “one example” of the present disclosure are not intended to be interpreted as excluding the existence of additional embodiments that also incorporate the recited features. The terms “first,” “second,” “third,” and so on are used merely as labels, and are not intended to impose numerical requirements or a particular positional order on their objects. The following claims particularly point out subject matter from the above disclosure that is regarded as novel and non-obvious.

Claims

Attorney Docket No. P240098WO CLAIMS:

1. A cognitive load management system of a vehicle, comprising: a processor, and a memory storing instructions that when executed, cause the processor to: during operation of the vehicle by a driver: estimate a cognitive load of the driver based on information received from a driver monitoring system (DMS) of the vehicle; generate a cognitive load score of the driver based on the estimated cognitive load; detect that the cognitive load score is below a first threshold value, and in response, engage the driver in conversation using a generative artificial intelligence (AI) model to increase the cognitive load.

2. The cognitive load management system of claim 1, wherein further instructions are stored in the memory that when executed, cause the processor to: detect that the cognitive load score is above a second threshold value, and in response, adjust one or more cabin controls of the vehicle to reduce the cognitive load.

3. The cognitive load management system of claim 1, wherein the information received from the DMS includes at least one of: output of a dashboard camera of the vehicle; seat occupancy data of the vehicle; and biometric data of the driver.

4. The cognitive load management system of claim 1, wherein further instructions are stored in the memory that when executed, cause the processor to: retrieve an interest profile of the driver from the memory, the interest profile including topics that the driver is interested in; submit a prompt to the generative AI model to initiate the conversation by asking a question to the driver based on the topics; convert the question outputted by the generative AI model into an audio stream using a text-to-voice converter of the cognitive load management system; and output the audio stream to a speaker of the vehicle.Attorney Docket No. P240098WO 5. The cognitive load management system of claim 4, wherein further instructions are stored in the memory that when executed, cause the processor to: select a topic of the interest profile based on the cognitive load score of the driver, according to a cognitive load mapping stored in the memory, the cognitive load mapping indicating topics of the interest profile that match the cognitive load score.

6. The cognitive load management system of claim 5, wherein the interest profile is generated from information about the driver extracted from a smart phone of the driver by a companion application of the cognitive load management system, the companion application installed on the smart phone.

7. The cognitive load management system of claim 6, wherein the information about the driver extracted from the smart phone is based on one or more of: one or more apps installed on the smart phone; preferences of an app installed on the smart phone; emails and / or text messages stored on the smart phone; histories of interactions with large language models (LLMs) using the smart phone; and preferences of the driver with respect to the cognitive load management system included in the companion application.

8. The cognitive load management system of claim 6, wherein the information about the driver extracted from the smart phone is used to classify the driver into a category, and the topic is selected based on the category.

9. The cognitive load management system of claim 4, wherein further instructions are stored in the memory that when executed, cause the processor to: receive an audio stream from the driver via a microphone of the vehicle, the audio stream including a response to the question; convert the received audio stream into text using the text-to-voice converter; enter the text into the generative AI model to continue the conversation.

10. The cognitive load management system of claim 1, wherein further instructions are stored in the memory that when executed, cause the processor to:Attorney Docket No. P240098WO prompt the generative AI model to gamify the conversation by providing challenge questions, and offering incentives as rewards for correct answers.

11. The cognitive load management system of claim 10, wherein the rewards include a free- of-charge use of a subscription service of the vehicle available to the driver for a specified duration.

12. The cognitive load management system of claim 10, wherein the challenge questions include questions based on statistical information collected by the cognitive load management system during a journey of the vehicle.

13. A method for a cognitive load management system of a vehicle, the method comprising: collecting in-cabin sensor information of a driver of the vehicle during operation of the vehicle, the in-cabin sensor information including data collected from a driver monitoring system (DMS) of the vehicle; estimating a cognitive load of the driver based on the collected in-cabin sensor information; generating a cognitive load score of the driver based on the estimated cognitive load; detecting that the cognitive load score is below a first threshold value, and in response, engaging the driver in conversation to increase the cognitive load using a generative artificial intelligence (AI) model.

14. The method of claim 13, wherein the in-cabin sensor information includes at least one of: an output of a dashboard camera of the vehicle; seat occupancy data of the vehicle; and biometric data of the driver.

15. The method of claim 13, wherein engaging the driver in conversation to increase the cognitive load using the generative AI model further comprises: retrieving an interest profile of the driver from a memory of the cognitive load management system, the interest profile including topics that the driver is interested in; submitting a prompt to the generative AI model to initiate the conversation by asking a question to the driver based on the topics;Attorney Docket No. P240098WO converting the question outputted by the generative AI model into an audio stream using a text-to-voice converter; outputting the audio stream to a speaker of the vehicle; receiving an audio stream from the driver via a microphone of the vehicle, the audio stream including a response to the question; converting the received audio stream into text using the text-to-voice converter; and entering the text into the generative AI model.

16. The method of claim 15, wherein submitting the prompt to the generative AI model to initiate the conversation further comprises: selecting a topic of the interest profile based on at least one of: a category that the driver is classified into by the cognitive load management system, based on the interest profile; the cognitive load score of the driver, according to a cognitive load mapping indicating topics of the interest profile that match the cognitive load score.

17. The method of claim 15, wherein the interest profile is generated from information about the driver extracted from a smart phone of the driver by a companion application of the cognitive load management system, the companion application installed on the smart phone.

18. The method of claim 13, further comprising: prompting the generative AI model to gamify the conversation by providing challenge questions and offering incentives as rewards for correct answers, the challenge questions including questions based on statistical information collected by the cognitive load management system during a journey of the vehicle, the rewards including a free-of-charge use of a subscription service of the vehicle available to the driver for a specified duration.

19. A method for a cognitive load management system of a vehicle, the method comprising: wirelessly receiving information from a companion application of the cognitive load management system installed on a smart phone of a driver of the vehicle, the information including one or more of the following: one or more apps installed on the smart phone; preferences of an app installed on the smart phone; emails and / or text messages stored on the smart phone; andAttorney Docket No. P240098WO histories of interactions with large language models (LLMs) using the smart phone; creating an interest profile of the driver based on the received wireless information, the interest profile including topics of interest of the driver; and engaging the driver in conversation based on a topic of interest of the interest profile, using a generative artificial intelligence (AI) model.

20. The method of claim 19, wherein the topic of interest is selected by the cognitive load management system based on an estimated cognitive load of the driver.

Images