Method for adapting an infotainment system, infotainment system and motor vehicle

Abstract

The invention relates to a method for adapting an infotainment system (35) comprised in a motor vehicle (32) to user preferences. To this end, a situational trigger (15) is detected and a point in time (16) at which the user (33) is susceptible to a prompt (11) is determined. Subsequently, a prompt (11) to the user (33) is constructed on the basis of the detected situational trigger (15) and the determined point in time (16), wherein the prompt (11) is intended to determine a user preference of the user (33). The user preference is subsequently determined on the basis of a reaction (23) of the user (33) to the prompt (11). Subsequently, a user profile is created on the basis of the determined user preference. Subsequently, a setting (38) of the infotainment system (35) is derived from the determined user preference and applied.

Description

Technical Field

[0001] The present invention relates to a method for adapting an infotainment system (Infotainment system) included in a motor vehicle to user preferences. Background Technology

[0002] Modern vehicle systems (such as voice dialogue systems in motor vehicles) typically rely on user-initiated input and are therefore static. If a dialogue initiated by the voice dialogue system (e.g., to determine user preferences) does not succeed directly, the user must repeatedly answer the same question or manually input to answer it.

[0003] US 2021 / 0 326 344 A1 discloses a virtual assistance system for a user, the virtual assistance system having a vehicle computing unit with the function of generating audiovisual questions based on context. These audiovisual questions are output based on a stored user profile and the current user context determined by sensors.

[0004] US 2022 / 0051669 A1 discloses an interactive assistance device for a user's vehicle navigation system, which has an active voice output function. To supplement the user profile, the voice output function asks the user questions related to previously recommended and / or determined function usage behaviors, depending on the context. Summary of the Invention

[0005] The purpose of this invention is to proactively and context-specifically suggest vehicle system functions that users rarely or never use.

[0006] This objective is achieved through the subject matter of the independent claims.

[0007] Advantageous improvements of the invention are described in the dependent claims, the following description, and the accompanying drawings.

[0008] This invention provides a method for adapting an infotainment system included in a motor vehicle to user preferences. The method includes the following steps:

[0009] First, one or more scenario triggers are detected, where such triggers can be, for example, environmental conditions, such as an external temperature below a preset threshold or within a preset range. For example, an external temperature of -5 to 15 degrees Celsius could represent a scenario trigger. Another scenario trigger could be a specific workday and / or month and / or season and / or specific location and / or network status, especially the battery status of the device or system and / or lighting conditions and / or specific weather conditions, such as, for example, snowfall. Therefore, if it is determined that it is currently snowing and / or the outdoor temperature is, for example, -5 degrees Celsius, then, for example, the infotainment system can automatically activate or suggest vehicle heating, for example, to 20 degrees Celsius, as an example only.

[0010] The determination of such situational triggers can be achieved through sensor devices, such as temperature sensors and / or snow depth sensors and / or light sensors and / or Global Navigation Satellite Systems (GNSS), all of which can utilize existing technologies. For example, to determine time variables (such as time and / or weekdays and / or months), the infotainment system can have a digital calendar system and / or a real-time clock (RTC) module.

[0011] Next, determine the time points when the user is likely to accept the prompts. The prompts can be (active) questions and / or (active) suggestions from the voice dialogue system. For example, such time points might occur when the user is currently in traffic congestion, and / or when the user's vehicle speed is within a preset range, such as, for example, 0 km / h to 30 km / h. That is, at such time points or moments, the user's attention is preferably not highly occupied. For this purpose, a speed sensor can be used to report such time points. Additionally or alternatively, such time points could be, for example, when the volume inside the vehicle is below a preset threshold (such as, for example, 10 dB to 65 dB), this can be considered a time point when the user is likely to accept the prompts.

[0012] Next, if at least one contextual cue and a time point at which the user is likely to accept the cue have been identified, a cue is constructed for the user. This cue aims to determine the user's preferences. For example, the voice dialogue system could output a cue such as, "What are your preferences for chair massage?" or "Would you like to try the 'wave' massage function?" Thus, the voice dialogue system can use this cue to suggest activating functions that the user might want to use.

[0013] Subsequently, user preferences are determined based on the user's response to the prompt. Therefore, the voice dialogue system can interpret the user's response, for example, using speech recognition. Additionally or alternatively, the vehicle may include an emotion recognition device configured to determine the user's response and / or forward it to the infotainment system. Thus, if a user requests a function suggested by the voice dialogue system (such as the massage function mentioned above), the voice dialogue system initiates the execution of that function. The voice dialogue system can then obtain some feedback from the user to understand their evaluation of the executed function. That is, if the user likes the function, they can manually or verbally convey their preference to the voice dialogue system, thereby storing the user's preferences.

[0014] User preferences may include at least one of the following: context triggers and / or the timing at which the voice dialogue system introduces and demonstrates a feature to the user via prompts; and / or the preferred date or time to use the feature; and / or the frequency of requesting the feature; and / or the duration of using the feature; and / or feedback in the form of star ratings; and / or voice comments given by the user after using the feature; and / or the reaction time between suggesting a feature through the voice dialogue system and the user's confirmation or rejection; and / or the emotion or mood detected in the user's voice; and / or pausing or switching to another feature during the use of the suggested feature; and / or improvements suggested by the user. For example, features may refer to: heated seats and / or adaptive lighting systems and / or car radios and / or automatic climate control and / or sliding sunroofs and / or parking assist systems and / or automatic start-stop systems and / or WLAN (wireless local area network) hotspots.

[0015] Subsequently, user profiles containing these user preferences are created. Based on these preferences, the infotainment system settings are derived or learned. Alternatively, it can be specified that only the settings are derived. For example, if a user preference includes the information "Activate seat heating when the outside temperature is below 10 degrees Celsius," then the function can be automatically activated when such an outside temperature is measured.

[0016] That is, the settings of the infotainment system are derived from the identified user preferences, especially when using user profiles, and these settings are applied.

[0017] The advantage of this invention is that it enhances user comfort because settings can be automatically adjusted based on user preferences. Furthermore, another advantage is that users only learn about new and / or unused vehicle features when they are readily accepted. This can positively impact the user experience. Additionally, the learning phase for creating user profiles can be shortened because the voice dialogue system does not need to passively observe user behavior but can proactively inquire about and save user preferences.

[0018] The invention also includes embodiments that provide additional advantages.

[0019] One improved approach specifies that prompts should be provided for functions that the user has not activated before receiving the prompt, and / or functions whose usage frequency is below a preset threshold. For example, the threshold could be in the range of 1 to 10. This can be achieved, for example, through a profiling tool that is part of the infotainment system. Users can then identify functions they do not use or rarely use.

[0020] One improvement specifies that usage frequency is determined by means of historical records, whereby the historical records quantify which functions of the motor vehicle have been activated and / or the frequency of activation of these functions during the period when a user is registered as an occupant. To register a user as an occupant, identification devices in the prior art, such as facial recognition systems and / or key recognition, can be used. Therefore, the historical records are a record or log of which functions of the motor vehicle have been used by the user and the frequency of their use.

[0021] One improved approach specifies that, in step a), the context trigger includes: the estimated travel time and / or the planned travel destination and / or the current outside temperature and / or the user's usage profile. As already described, this information (e.g., the estimated travel time) can be read by sensors, in particular by a global navigation satellite system. For example, if the estimated travel time exceeds ten minutes, and if there are also times when the user is readily aware of the prompt, the user can be asked, for example, whether they would like to listen to a podcast.

[0022] One improved approach specifies that, in step b), the timing is determined based on the vehicle's speed and / or the volume of the interior space and / or the determined number of occupants in the vehicle and / or the surrounding traffic density. For example, if the infotainment system identifies that the user is in traffic congestion via environmental sensors, particularly ultrasonic and / or radar sensors, it can prompt the voice dialogue system to generate a prompt to the user. The advantage of this is that prompts are delivered to the user only in situations where at least a low level of inattention is present.

[0023] One improved approach specifies that step c) is based on "active question-and-answer" voice dialogue, wherein the voice dialogue is initiated by a voice dialogue system and engages in conversation with the user. The voice dialogue system is designed to receive voice and / or text-based and / or menu-based feedback from the user. For implementation of voice dialogue, those skilled in the art can refer to the publication "Ask the Right Questions: Active Question Reformulation with Reinforcement Learning" by Buck, Christian, et al.

[0024] In response, one improved approach specifies that the voice dialogue system possesses an artificial neural network, trained using reinforcement learning on a dataset (experience dataset) formed from question-answer pairs, to construct prompts based on at least one contextual trigger and / or determined time points. Here, the artificial neural network refers to software capable of comprehensively implementing voice dialogue within the voice dialogue system.

[0025] One improved approach stipulates that steps b) to e) are skipped once the artificial neural network reports that a preset threshold for the context trigger has been reached. For example, if the neural network learns a confidence level or percentage for a context trigger that is between 80% and 90%, steps b) to e) are skipped. For instance, the neural network might associate an 80% confidence level for a specific external temperature (e.g., -5°C to 5°C) as the context trigger being off, thus eliminating the need to ask the user whether the seat heating should be activated; the function should now be executed automatically. For example, if the user subsequently manually deactivates the seat heating, this learned context trigger can be at least iteratively discarded. That is, the confidence level will be below 80% the next time the context trigger occurs.

[0026] One improved approach specifies that step c) includes reconstructing the question if user preferences cannot be determined and / or if user confusion gestures are determined. Reconstruction involves proposing a new question directly following the initially constructed question, based on an ordered probability list derived from the results of the voice dialogue. Therefore, the reconstructed question is based on an ordered probability list derived from the results of the previous voice dialogue, with the new question immediately following the initially proposed question. This ensures reduced misunderstanding and / or reliable determination of user preferences.

[0027] One improved embodiment specifies that the confusion gesture includes, through image-based and / or radar-based and / or lidar-based and / or microphone-based user monitoring, recognizing as a response to a prompt issued by the infotainment system, the user shrugs and / or frowns and / or groans and / or performs a predetermined preset gesture for seeking help.

[0028] One improved approach specifies that, as a response, detection includes: verbal feedback and / or tactile feedback and / or gestures captured by a camera (such as a nod), and / or the length of time elapsed (i.e., the length of time the cue is ignored). This length of time can be, for example, 10 to 30 seconds. Additionally or alternatively, as a response, detection may include: changes in user settings and / or emotional responses. Changes in user settings may be, for example, muting or reducing the volume of the infotainment system, for example, reducing it by 3 to 10 units or by 10 to 20 decibels. Emotional responses can be detected using biosensors (such as skin conductance sensors) and / or facial recognition software for recognizing micro-expressions and / or thermal imaging cameras for measuring changes in facial temperature.

[0029] Additionally or alternatively, tone of voice can be detected via microphone, allowing the frequency variations in the tone at the end of words (e.g., distinguishing between "rising," "falling," and "remaining unchanged") to be fed into a neural network as input data. A particular advantage of combining facial expressions and tone of voice is the ability to detect audiovisual signals to interpret the user's emotional responses.

[0030] One improved approach specifies that the activation or deactivation of the derivation of at least one setting can be controlled based on manual input or feedback. In other words, users can selectively control or disable the infotainment system's learning process. That is, if the user triggers the initiation of this learning process, it can be specified that user preferences are determined based on detected or pending contextual triggers. For example, the activation of the learning process can be configured such that the infotainment system should detect user preferences for the next half hour, the next two hours, or (temporarily) without time limit. Similarly, it can be specified that user preferences should not be determined for the next half hour, the next two hours, or (temporarily) without time limit. Thus, users can control when user preferences should be detected. Furthermore, concerns about data protection can be reduced. For example, in a trip with multiple passengers, different music styles may be played. Therefore, it is meaningful to be able to manually deactivate the learning process to prevent unfamiliar music preferences from being attributed to a user's preferences.

[0031] One improvement stipulates that the transparency of settings is ensured, at least in part, by displaying the derived settings in the AI ​​menus or user interface of the infotainment system. In other words, for a given setting, it can be disclosed from which user preferences it was derived. Transparency can be ensured, for example, by displaying history or logs, allowing users to read the frequency of use of a feature and / or a context trigger. This allows users to (better) understand their own habits and / or preferences. When "mislearned" settings are discovered (i.e., settings the user dislikes), the reasons can be investigated. By ensuring this transparency, users are more likely to build trust in the infotainment system, thereby improving user-friendliness.

[0032] For application scenarios or situations that can be obtained in this method but are not explicitly described herein, it may be specified that error reports be output and / or user feedback be requested according to this method, and / or default settings and / or predetermined initial states be set.

[0033] The present invention also includes an infotainment system for a motor vehicle. The infotainment system may have a data processing device or a processor device configured to perform embodiments of the method according to the invention. For this purpose, the processor device may have at least one microprocessor and / or at least one microcontroller and / or at least one FPGA (Field-Programmable Gate Array) and / or at least one DSP (Digital Signal Processor). As a microprocessor, a CPU (Central Processing Unit), GPU (Graphics Processing Unit), or NPU (Neural Processing Unit) may be used, in particular. Furthermore, the processor device may have program code configured to perform embodiments of the method according to the invention when executed by the processor device. The program code may be stored in the data memory of the processor device. For example, the processor device may be based on at least one circuit board and / or at least one SoC (System-on-Chip).

[0034] The present invention also includes improvements to the infotainment system and / or motor vehicle according to the invention, having the features described in the improvements already incorporated with the method according to the invention. For this reason, the corresponding improvements will not be described further herein.

[0035] Preferably, the motor vehicle according to the invention is designed as an automobile, especially as a passenger car or commercial vehicle, or as a bus or motorcycle.

[0036] As another solution, the invention also includes a computer-readable storage medium comprising program code that, when executed by a computer or group of computers, causes implementation of an embodiment of the method according to the invention. The storage medium may be provided at least partially as non-volatile data storage (e.g., flash memory and / or SSD - solid-state drive) and / or at least partially as volatile data storage (e.g., RAM - random access memory). The storage medium may be arranged in a computer or group of computers. However, the storage medium may also operate on the Internet, for example, as a so-called app store server and / or cloud server. A processor circuit having, for example, at least one microprocessor may be provided by the computer or group of computers. The program code may be provided as binary code and / or as assembly code and / or as source code in a programming language (e.g., C) and / or as a program script (e.g., Python).

[0037] The present invention also includes feature combinations of the described embodiments. Therefore, the present invention also includes implementations having various feature combinations of the described embodiments, provided that these implementations are not described as mutually exclusive. Attached Figure Description

[0038] Embodiments of the present invention are described below. Wherein:

[0039] Figure 1 A system for performing the method according to the invention is shown;

[0040] Figure 2 The technical implementation scheme for performing the method according to the invention is shown; and

[0041] Figure 3 An embodiment of the invention according to the present invention is shown in a motor vehicle. Detailed Implementation

[0042] The embodiments explained below are preferred embodiments of the present invention. In the embodiments, each described part constitutes a single, independent feature of the present invention, and these features also independently improve the present invention. Therefore, this disclosure should also include feature combinations different from the feature combinations of the illustrated embodiments. Furthermore, the described embodiments can also be supplemented by other features among the features already described in the present invention.

[0043] In the accompanying drawings, the same reference numerals denote elements that have the same function.

[0044] Figure 1A system for implementing the described concept is shown. This system can consist of two parts. One part is the existing geni:OS 14 (see also https: / / www.semvox.de / technologien / genios / ), and the other part is the question-answering model 13 integrated therein. A geni:OS IQA agent 1 that interacts with the following components can be integrated into geni:OS 14: these components could be the described sensor 6 and / or dialogue history or interaction history or log 4 and / or missing information 12. This missing information 12 can be obtained from the central user profile 3. In other words, when missing information 12 is identified, it can be extracted from the central user profile 3. For example, information that might be important for constructing prompt 11 is what music user 33 likes to listen to, thus this information can be obtained from the central user profile 3.

[0045] Each of these components 6, 4, and 12 can be connected to encoder networks 2, 2', or 2'', respectively, thereby generating so-called hidden states. These can then be cascaded (Konkatenation 8) to precisely integrate these hidden states. Additionally or alternatively, it can be specified that the geni:OS IQA agent 1 performs language style adaptation 9 from the dialogue history 4 and / or missing information 12—for example, by means of a neural style transfer model. This language style adaptation 9 can be added to cascade 8, thereby adjusting the content of cascade 8 through language style adaptation 9. This content can then be fed to decoder network 10. Generally, it is specified that the question-answering model or artificial neural network 13, as described, is trained using reinforcement learning on a dataset (empirical dataset) formed from question-answer pairs from question corpus 5, enabling it to construct prompts or questions 11 based on at least one contextual trigger 15 and / or determined time points 16. The created prompts 11 can be sent to the geni:OS IQA agent 1, which then plays them as speech output. Therefore, geni:OS IQA agent1 can refer to a specific voice dialogue system.

[0046] Figure 2The technical implementation of this concept is illustrated. For example, the described context trigger 15 can be identified, and / or reported via tactile and / or voice interaction. Furthermore, the time point 16 at which the user 33, as described above, is receptive to the prompt 11 can be determined using the various sensors 6 mentioned above. These sensors 6 can determine and / or assess the user 33's mood and / or burden and / or fatigue level. Data fusion 17 can be performed based on the determined time point 16 and by adding user preferences from the central user profile 3. This data fusion 17 can include the dialogue history 4 and / or a set of rules 18 for performing the data fusion 17. A result 19 can then be derived. This result 19 can be specified to be checked against an integrity criterion, thus allowing for a report of completeness or incompleteness. For example, the integrity criterion could be a confidence level achieved, which can be calculated or read from the dialogue history 4. Therefore, if the confidence level is below a preset threshold, the voice dialogue system 1 can be prompted to construct the prompt 11. For this purpose, the prompt 11 can be constructed, for example, by obtaining and / or presenting existing questions from a question corpus 5 containing question-answer pairs. In response to prompt 11, language style adaptation 9 can be performed. The result can be a reconstructed question 24, upon which a question-and-answer dialogue 25 can then be initiated. Subsequently, this question-and-answer dialogue 25 can be evaluated 28.

[0047] However, if the confidence level is higher than a preset threshold, this could mean that the learned action 21 should be executed. At this point, a confirmation prompt 22 can be issued to the user 33. Subsequently, the user 33's response 23 can be determined or detected. Next, the interaction can be evaluated 28. If the confidence level of this evaluation 28 is lower than the threshold, the result is considered incomplete, and this signal is sent to the reconstruction model 30 to create a reconstructed question 31. The reconstructed question 31 can be, for example, an explanation, recommendation, or follow-up question. Based on the reconstructed question 31, a question-and-answer dialogue 25 can be conducted again, whereby the dialogue is evaluated again 28. The question-and-answer dialogue 25 can be based on an NLU (Natural Language Understanding) corpus 26 and / or on emotion recognition 27 and / or on dialogue history or interaction history 4. If the result is considered complete, the interaction can be learned, refined, or even discarded as an action, thereby adapting the AI ​​interaction 29.

[0048] Figure 3A user 33 in a motor vehicle 32 is shown. The motor vehicle 32 includes an infotainment system 35 with a processor 36. Furthermore, the infotainment system 35 includes a voice dialogue system 1. Additionally, the motor vehicle 32 has a temperature sensor 6. An external temperature 37 of -5 degrees Celsius is symbolically shown. This external temperature 37 can be detected by the temperature sensor 6 in the infotainment system 35 as a context trigger 15. Next, a time point 16 at which the user 33 is receptive to a prompt 11 can be determined. If this time point 16 is determined, a prompt 11 for the user 33 can be constructed based on the detected context trigger 15 and the determined time point 16, wherein the prompt 11 aims to determine the user 33's user preferences. The prompt 11 is symbolically represented by a question mark. User preferences can be determined based on the user 33's response 23 to the prompt 11. Based on user preferences, a user profile can be created. Subsequently, settings 38 of the infotainment system 35 can be derived from the determined user preferences and applied.

[0049] In general, the examples demonstrate how an inverse question answering (IQA) voice dialogue system can be provided.

[0050] List of reference numerals in the attached diagram:

[0051] 1. Voice dialogue system

[0052] 2 encoder module 1

[0053] 2' Encoder Module 2

[0054] 2'' Encoder Module 3

[0055] 4. Dialogue History

[0056] 5. Problem Corpus

[0057] 6 sensor devices

[0058] 7 edge endpoints

[0059] 8-level cascade

[0060] 9 Language style adaptation

[0061] 10 decoders

[0062] 11 Tips

[0063] 12 Missing Information

[0064] 13 Reverse QA Model

[0065] 14 geni:OS Human-Powered Intelligent Interaction

[0066] 15 Contextual Triggers

[0067] 16 time points

[0068] 17 Data Fusion

[0069] 18 Rules Set

[0070] 19 Results

[0071] 20 Reverse QA

[0072] 21 learned actions

[0073] 22 Confirmation Prompt

[0074] 23 reactions

[0075] 24 Reconstruction Issues

[0076] 25 Questions and Answers

[0077] 26 NLU Corpus

[0078] 27. Emotion Recognition

[0079] 28 assessments

[0080] 29 Adaptation to AI Interaction

[0081] 30 Reconstruction Model

[0082] 32 motor vehicles

[0083] 33 users

[0084] 35 Infotainment System

[0085] 36 processor devices

[0086] 37 external temperature

[0087] 38. Settings.

Claims

1. A method for adapting an infotainment system (35) included in a motor vehicle (32) to user preferences, wherein, The method includes: a) Detection context triggers (15) b) Determine the time point (16) when the user (33) is more receptive to the prompt (11). c) Based on the detected context triggers (15) and the determined time points (16), a cue (11) is constructed for the user (33), wherein the cue (11) aims to determine the user preferences of the user (33). d) Based on the user's (33) response to the prompt (11) (23): determine user preferences, e) Create user profiles based on the identified user preferences. f) Derive the settings (38) of the infotainment system (35) from the determined user preferences and apply the settings (38).

2. The method according to claim 1, characterized in that, The prompt (11) is for functions that have not been activated before the time (16) when the user (33) receives the prompt (11), and / or for functions that the user (33) uses less frequently than a preset threshold.

3. The method according to claim 2, characterized in that, The frequency of use is determined by means of historical records (4), which quantifies which functions of the vehicle (32) have been activated and / or the frequency of activation of these functions in the past during the period when the user (33) was registered as a passenger.

4. The method according to any one of the preceding claims, characterized in that, In step a), the scenario trigger (15) includes: the expected driving time and / or the planned driving destination and / or the current outside temperature (37) and / or the user's (33) usage profile.

5. The method according to any one of the preceding claims, characterized in that, In step b), the time point (16) is determined based on the driving speed of the motor vehicle (32) and / or the volume of the interior space and / or the determined number of occupants in the motor vehicle (32) and / or the surrounding traffic density.

6. The method according to any one of the preceding claims, characterized in that, Step c) Based on “active question and answer” voice dialogue (25), wherein the voice dialogue is initiated by the voice dialogue system (1) and conducted with the user (33).

7. The method according to claim 6, characterized in that, The voice dialogue system (1) has an artificial neural network and is trained on a dataset formed by question-answer pairs by means of reinforcement learning, enabling it to construct prompts (11) based on at least one contextual trigger (15) and / or a determined time point (16).

8. The method according to claim 6 or 7, characterized in that, Once the artificial neural network report has reached the preset threshold of the context trigger (15), skip step b) to e).

9. The method according to any one of the preceding claims, characterized in that, Step c) includes reconstructing the question if user preferences cannot be determined and / or if user (33)’s confused gestures are determined, wherein reconstructing includes proposing a question directly following the initially constructed question based on an ordered probability list of the voice dialogue results.

10. The method according to any one of the preceding claims, characterized in that, Confusion gestures include, through image-based and / or radar-based and / or lidar-based and / or microphone-based user monitoring (6), recognizing as a response (23) to a prompt (11) issued by the infotainment system (35), the user (33) shrugs and / or frowns and / or groans and / or performs a predetermined preset gesture for seeking help.

11. The method according to any one of the preceding claims, characterized in that, As a reaction (23), at least one of the following is detected: - Verbal feedback and / or tactile feedback, - Gestures, - The length of time elapsed - Changes in user settings (38), - Emotional reaction.

12. The method according to any one of the preceding claims, characterized in that, The activation or deactivation of at least one setting (38) is controlled by manual input.

13. The method according to any one of the preceding claims, characterized in that, The derived settings are made transparent, at least in part, by displaying the derived settings (38) in the AI ​​menu of the infotainment system (35).

14. The infotainment system (35) according to any one of the preceding claims, characterized in that, The infotainment system (35) has a processor device (36) having program instructions that, when executed by the processor device (36), cause the processor device to perform the method according to any one of the preceding claims.

15. A motor vehicle (32) having an infotainment system (35) according to claim 14.

Images