METHOD FOR OPERATING AN INTERACTIVE INFORMATION SYSTEM FOR A VEHICLE, AND A VEHICLE

Description

[0001] The invention relates to a method for operating an interactive information system for a vehicle and a vehicle.

[0002] The connected vehicle, which communicates with its immediate surroundings and uses the internet for this purpose, can already be considered state-of-the-art. In this context, corresponding dialogue systems in the vehicle itself are known, which also enable the driver or passengers to interact with the vehicle or, via the vehicle, with their immediate surroundings.

[0003] In-vehicle voice dialogue systems are becoming increasingly important for automakers because, in an age of automated driving functions, they create an additional driving experience that involves the ability to communicate with the vehicle. This includes, for example, obtaining information about current events or asking general questions about the driving situation. For example, while driving, passengers might wonder which building is currently within sight of the vehicle. Such a question could then be addressed directly to the vehicle itself via voice input. The vehicle, or a corresponding associated system, could then answer this and other similar questions appropriately.

[0004] In this context, initial scientific publications have been published that address the topic of "question-answering" (question-answering = questions - answers) and language processing in general. Context analysis considers an expanded approach. The basic idea of context analysis is to use spoken dialogues between a system and user in a data-driven approach to recognize when the user's question is connecting to a previously posed question or when the question introduces a new topic. For example, after the system has already provided an answer to a first question, it can recognize that a possible subsequent question relates to the context of the first question and can therefore tailor a second answer and its content accordingly.In other words, the last question refers to the question asked previously, without the user having to create a direct link in terms of content or facts.

[0005] Initial approaches to solutions are already known from the state of the art.

[0006] From the document DE 10 2015 210 430 A1, a method for recognizing a speech context for a voice control system and a method for determining a speech control signal for a voice control system and a device for carrying out the methods are known. The method comprises a step of reading in viewing direction information about a current viewing direction of a vehicle occupant, a step of assigning the viewing direction information to a viewing zone in an interior of the vehicle in order to obtain viewing zone information about a viewing zone currently viewed by the occupant, and a step of determining speech context information about a predetermined speech context assigned to the currently viewed viewing zone using the viewing zone information.

[0007] From the document DE 10 2015 100 900 A1, setting up a speech recognition system using context information is known. One embodiment provides a method comprising the following steps: obtaining context information relating to an information processing device using a processor. Subsequently, adapting an automated speech recognition engine using the context information is performed using a processor. This is followed by receiving a user speech input at an audio receiver of the information processing device, and finally, the step of providing, using a processor, a recognized speech based on the received user speech input and adapting the context information to the automated speech recognition engine.

[0008] DE 10 2014 118 450 A1 discloses an audio-based system and method for classifying in-vehicle context as known. Each context corresponds to one or more events associated with the vehicle, for example, the radio being turned on and / or a window being opened. The method includes detecting sound activity in an audio signal detected in the vehicle and assigning a context to the vehicle based on the detected sound activity.

[0009] Non-audio data such as the operational status of a vehicle system or device is used to assign context.

[0010] Furthermore, DE 10 2008 062 542 A1 discloses in-vehicle, circumstantial voice recognition. A variety of parameters associated with a variety of vehicle functions are monitored as an indicator of the current vehicle conditions. At least one vehicle function is identified as a candidate for intended user control based on user interaction with the vehicle. The identified vehicle function is then used to distinguish between potential commands contained in speech received from the user.

[0011] From the document EP 2 211 336 A1 a method for recognizing the voice input of a user can be taken, wherein the following steps are provided: recognizing the voice input of the user, recognizing the voice input of the user by comparing the voice input with a list of entries using language model statistics in order to determine the language which best matches the voice input of the user, capturing navigation information of a journey to a predetermined destination, wherein the most probable entry is determined by modifying the language model statistics taking into account the navigation information.

[0012] A personal vehicle assistant is also known from the publication US 2014 / 136 187 A1. In particular, a personal vehicle assistant is disclosed that engages a user in a conversational dialogue about vehicle-related topics, such as those typically found in a vehicle's owner's manual. Such a personal vehicle assistant includes modules for interpreting spoken natural language input, searching a vehicle knowledge database and / or other data sources for relevant information, and responding to the user's input in a conversational format. The dialogue can be initiated by the user or more proactively by the personal vehicle assistant based on events that may currently be occurring with the vehicle.The personal vehicle assistant can use real-time inputs received from the vehicle and / or non-verbal inputs from the user to improve its understanding of the dialogue and assist the user in a variety of ways.

[0013] A voice and connection platform is also known from US 2016 / 098 992 A1. In particular, a system and method for providing a voice assistant are disclosed. The method comprises the following steps: receiving a first audio input from a user requesting a first action on a first device; performing automatic speech recognition on the first audio input; obtaining a user context; performing natural language understanding based on the speech recognition of the first audio input; and performing the first action based on the user context and the natural language understanding.

[0014] Today's systems often focus on a specific aspect and frequently already utilize voice input functions for this purpose. A complete integration of as many available information sources as possible is neglected.

[0015] The invention is based on the object of providing a method for operating an interactive information system for a vehicle and a vehicle, which ensures user-friendly operation and a high information content.

[0016] In a preferred embodiment of the invention, a method for operating an interactive information system for a vehicle is provided. Such a method comprises the following steps: operating a context information system which is designed to receive and process a first request from a user via an input unit such that at least a first response is generated and made available via an output unit. A second request, which is made directly after the first request, is prepared in the context of the first request using a context analysis before being processed by the context information system such that at least a second response is generated and made available via an output unit. Depending on the preparation, the response is given either in the context of the first response or independently of the first response.The user's first and / or second query relates to an external object in the vicinity of the vehicle. In addition, at least one storage unit is operated, which can be coupled to the context information system and is designed to store activities of the context information system and make them available to the context information system for the purpose of processing queries. In addition, at least one component of a vehicle sensor system is operated, which can be coupled to the context information system and is designed to capture and process at least one piece of information from the immediate and / or extended surroundings of the vehicle and / or from the vehicle itself. This at least one piece of information is made available to the context information system, so that a respective response to a user's query is additionally processed in the context of this information and is made available via an output unit.In this way, not only user-friendly operation but also a high information content can be ensured. The context information system can be structured along the lines of already established systems or at least partially be a conventional system, such as a mobile application or an on-board computer. The activities of the context information system can include any interactions during a dialogue that are considered useful for a current and / or subsequent query. The process is not significantly limited at this point, and in extreme cases, it can lead to, for example, too much information being stored and / or utilized across the various registration levels rather than too little.Activities could also include a time-dimensional classification of queries from one or more users, taking into account the current date input as well as the fact that the vehicle, for example, is moving through different time zones during the journey or that a query refers to different time zones. Known systems often have names that either represent a person's proper name or are derived from such a proper name. Such systems are also known under the term "Relevancy Recognition for Contextual Question Answering," where the focus is on the context analysis itself.The basic idea is to use the spoken dialogues between system and user in a data-driven approach to recognize when the user's question is following up on a previously asked question and when the question introduces a new topic. In other words, a recently asked question may refer to a previously asked question. The system must recognize this and provide an answer without the user having to provide additional explanations. These systems already have a self-contained logic that is suitable for a dialogue with a user. The presented method goes a step further by coupling this logic with additional information and then blending it to produce an even higher information content.This creates a special context model with the user, which collects, evaluates, and classifies a multitude of information, and can output a more complex response. All information from a vehicle sensor system, such as information about a current vehicle journey, can be captured by the individual components of this system and either utilized directly or initially stored for subsequent use or application in a respective dialogue with the user. Links can therefore not only be created in the context information system according to existing logic, but can also be additionally evaluated and classified based on an expanded information base, and thus output as a prepared response with more complex content.In other words, all information collected using the presented method is not only listed and stored, but also further aggregated and linked using the interactive information system, allowing queries from one or more users to be answered more comprehensively. For example, all information is technically stored in a so-called knowledge graph, so that user queries are translated into a query language suitable for knowledge databases and applied. Thus, using the presented method, relevant information can be continuously collected and stored ready for retrieval, for example, directly in the vehicle.The interactive information system can have appropriate interfaces and control units, or existing capacities and components in the vehicle, such as parts of an on-board computer or a vehicle control system, can be used accordingly. In this sense, the interactive information system is both a kind of platform and interface, allowing the various, initially independent systems to be combined in a suitable manner into a new, overarching logic. In this sense, interaction therefore not only concerns the actual communication with one or more users, but also indicates this interaction between the otherwise independent individual systems, thus enabling a meaningful and user-friendly response in the form of the desired answers.The components of the vehicle sensor system can, for example, include all control units and sensor devices, so that the interactive information system can ultimately utilize a multitude of combination options for the proposed method. For example, any distance sensors and associated controllers can be used with their respective information to provide the user with a higher level of information content through appropriate blending within the meaning of the method.

[0017] In a further preferred embodiment of the invention, a vehicle is provided which is designed to operate a method according to claims 1 to 8 before, after, or during a journey. The aforementioned advantages also apply to the presented vehicle, to the extent that they are transferable.

[0018] Further preferred embodiments of the invention emerge from the remaining features mentioned in the subclaims.

[0019] Thus, in a further preferred embodiment of the invention, the input unit is a voice-controlled input unit which is designed to receive and process a voice input, and / or wherein the input unit is designed to receive and process digitally processed input information. This ensures a particularly user-friendly method since, for example, a user can submit queries using voice alone. Additionally or alternatively, the user can also provide digitally processed input information. This could be done, for example, using a keyboard in the vehicle and / or a touch-sensitive screen. Also conceivable is input via short message or, in general, input via a computing unit which is linked in any form to the interactive information system and / or to the vehicle.At this point, there are no significant limits to be seen as long as an editable input is ultimately carried out, for example in digital form in the sense of an at least partially standardized arithmetic unit language or dialect.

[0020] In a further preferred embodiment of the invention, the output unit is a voice-controlled output unit designed to output a voice output, and / or the output unit is designed to output digitally processed output information. This allows an even more user-friendly design of the method. A user can thus, for example, take note of a response while performing another activity. The response could also be received with a time delay, independently of the vehicle, via a mobile device in the form of text or image. No significant limits can be seen here, as long as a processable input is ultimately carried out, for example in digital form in the sense of an at least partially standardized arithmetic unit language or dialect.

[0021] Furthermore, in a further preferred embodiment of the invention, it is provided that the user is a driver and / or passenger of the vehicle, wherein a voice of the user is recognized by the context information system and a personal knowledge base stored in the memory unit is taken into account when answering for this user. In this way, an answer can be prepared even better for a desired context and content and then output. For example, a link can be created with previously stored information about a specific user, enabling a targeted answer that is tailored, for example, to the user's preferences. For example, a journey in the city could have an open restaurant in the immediate vicinity as the destination.A user's verbal inquiry, via speech and voice recognition, leads the system to use previously stored information about a favorite restaurant, for example, to point out that it only opens at a certain time or is unfortunately closed today. Such a dialogue could develop its own dynamic without requiring all the detailed questions to be asked beforehand. This process thus makes a user-friendly application very easy to implement.

[0022] Furthermore, the method is provided for the following further step: operating at least one component of a user sensor system, which is designed to capture and process at least one piece of information about the user, so that this at least one piece of information is made available to the context information system, so that a respective response to a user query is additionally processed in the context of this information and made available via an output unit. This could, for example, be a component that registers a user's field of vision detection and then uses this resulting information to provide a more meaningful response to a query. For this purpose, the driver's head position, gestures, and facial expressions are recorded.If the driver looks in a certain direction and also asks about a building, for example, the answer is processed in such a way that the system not only logically associates the context of the question with this direction of view, but then provides a more meaningful answer. This allows the user to operate such a process comfortably, as minimal input still leads to a logical and complex answer. The recorded information is also interpreted by the information system to indicate a viewing or pointing angle, for example of an arm or a hand, depending on the current GPS position, which limits the area in which to search for relevant objects on the map, for example. All moving objects in the vicinity of the vehicle in which the process is applied, such as other vehicles and bicycles, are also recorded and used accordingly in a context-based dialogue.A query about a red vehicle could then logically be interpreted as a sports car located to the side, with an index finger pointing at this sports car, for example, helping the system to provide a more precise and meaningful answer. Furthermore, temporal validity of a query can be taken into account. For example, due to system latencies and the movement of the vehicle, the query may not be understood by the system until the user has already passed the object of interest. Such a circumstance can be additionally factored out by the system, which, for example, does not only occur based on the component of a user sensor system, but in particular based on the interaction with components of the vehicle sensor system in conjunction with components (or just one) of the user sensor system.

[0023] A further preferred embodiment of the invention also provides that activities of the context information system for each vehicle journey are stored as a self-contained knowledge base, so that responses to queries are prepared and output within the context of this knowledge base. For example, information relevant to the vehicle context is continuously recorded and is available in the vehicle for retrieval. This information can include the vehicle position, direction of travel, and information on sights or points of interest within a defined radius around the vehicle. A query can be analyzed within the context of this current journey and all actions already performed by the information system during this journey and answered accordingly.Technically, this could be solved, for example, in the form of a so-called knowledge graph about this journey, which is continuously stored and made available for processing queries. Such a knowledge graph can be translated into a query language suitable for knowledge databases. This makes it possible to provide a particularly user-friendly process, as it ensures even more targeted and meaningful answers to queries.

[0024] Furthermore, in a further preferred embodiment of the invention, it is provided that the context information system, the vehicle sensor system, the user sensor system, and the at least one storage unit and / or respective components of these systems are designed to be connected to the Internet. The aforementioned advantages can thus be achieved even more effectively, since, in addition to the existing resources and data, additional sources can be accessed via the Internet to answer queries. At least parts of the method can also be carried out via the Internet on spatially remote storage units and computing units, so that a user only requires a terminal infrastructure to use the information system itself.

[0025] Furthermore, in a further preferred embodiment of the invention, the component of a vehicle sensor system is selected from: front radar information device, rear radar information device, ultrasonic side detection device. These devices can individually comprise associated technical components that allow both the respective distance information to be collected and this information to be provided in appropriate processing in accordance with the method. These components can also be used in combination, so that a user query can be processed accordingly in the context of this individual or combined information. The aforementioned advantages can thus be achieved even more effectively.

[0026] Finally, in a further preferred embodiment of the invention, the component of a user sensor system is selected from: vehicle interior camera system, infrared sensor device, motion sensor device, 3D camera system. In this way, user inquiries in the vehicle interior can be processed even more effectively and with the additional information collected by these devices individually or in combination, thus enabling even more meaningful responses. For example, a user's emotional states can be registered and classified in this way. This analysis or measurement can thus be placed in the context of a question in order to provide a logical and expanded answer.For example, when characteristic moods of the user are detected, their previously stored tastes and preferences can be used to better process inputs or requests. The 3D camera system can, for example, be designed with at least two camera devices, ensuring particularly good spatial detection of the interior and the user. Using the components listed alone or in combination, a field of vision can be recognized even more accurately. User actions, such as a strongly gesticulating hand, can thus be better recognized and interpreted accordingly. User emotions can also be better analyzed. Using the infrared sensor device, for example, the surface of the skin can be analyzed, allowing additional information to be collected and used appropriately. The aforementioned advantages can thus be achieved even more effectively.

[0027] The various embodiments of the invention mentioned in this application can be advantageously combined with one another, unless otherwise stated in the individual case.

[0028] The invention is explained below in exemplary embodiments with reference to the accompanying drawings. They show: Figure 1 shows a process diagram of a method for operating an interactive information system for a vehicle; Figure 2 shows a vehicle designed to operate a method according to claims 1 to 8 before, after, or during a journey.

[0029] Figure 1shows a process diagram 10 of a method for operating an interactive information system for a vehicle 18. In a first step 12, a context information system is operated which is designed to receive and process a first request from a user via an input unit, such that at least a first response is generated and made available via an output unit, wherein a second request, which is made immediately after the first request, is prepared in the context of the first request before being processed by means of the context information system, such that at least a second response is generated and made available via an output unit, wherein, depending on the preparation, the response is either given in the context of the first response or is given independently of the first response.In a second step 14, at least one storage unit is operated, which can be coupled to the context information system and is designed to store activities of the context information system and make them available to the context information system for the purpose of processing queries. In a third step 16, at least one component of a vehicle sensor system is operated, which can be coupled to the context information system and is designed to capture and process at least one piece of information from the immediate and / or extended surroundings of the vehicle 18 and / or from the vehicle 18 itself, such that this at least one piece of information is made available to the context information system, such that a respective response to a user query is additionally processed in the context of this information and is made available via an output unit.The individual steps of the process diagram 10 can partially overlap and, in embodiments not shown in detail, can also be carried out in parallel.

[0030] Figure 2 shows a vehicle 18 designed to operate a method according to claims 1 to 8 before, after, or during a journey. For this purpose, an interactive information system 20 for a vehicle 18 is shown in the storage area 22 at the front of a windshield 24. This interactive information system 20 may have been placed there by a driver 26 or may itself be part of an interior fitting of the vehicle 18 (not shown in detail). List of reference symbols

[0031] 10Process diagram 12First step 14Second step 16Third step 18Vehicle 20Interactive information system 22Storage 24Windshield 26Driver

Claims

1. A method for operating an interactive information system (20) for a vehicle (18), comprising the following steps: • Operating a context information system that is designed to receive and process a first inquiry from a user via an input unit, so that at least a first response is generated and made available via an output unit, wherein a second inquiry, which is made immediately after the first inquiry, is prepared in the context of the first inquiry by means of a context analysis before being processed by means of the context information system, so that at least a second response is generated and made available via an output unit, wherein, as a function of the preparation, the second response is given either in the context of the first response or is given independently of the first response, and the first and / or second inquiry of the user relate to an external object in the vicinity of the vehicle (18), • Operating at least one storage unit that can be coupled to the context information system and is designed to store activities of the context information system and to make them available to the context information system for the purpose of processing inquiries, • Operating at least one component of a vehicle sensor system that can be coupled to the context information system and that is designed to capture and process at least one first item of information with respect to the external object in the vicinity of the vehicle (18), so that the at least one first item of information is made available to the context information system, so that a particular response to the user's inquiries is additionally processed in the context of the first item of information and is made available via an output unit, and • Operating at least one component of a user sensor system that is designed to capture and process at least one second item of information about a viewing direction, head position and / or gesture of the user that can be associated with the external object in the vicinity of the vehicle (18), so that the at least one second item of information is made available to the context information system, so that a particular response to the user's inquiries is additionally processed in the context of the second item of information and is made available via an output unit.

2. The method according to claim 1, wherein the input unit is a voice-controlled input unit that is designed to receive and process a voice input, and / or wherein the input unit is designed to receive and process digitally processed input information provided by the user.

3. The method according to one of the preceding claims, wherein the output unit is a voice-controlled output unit that is designed to output a voice output, and / or wherein the output unit is designed to output digitally processed output information to an external terminal.

4. The method according to one of the preceding claims, wherein the user is a driver (26) and / or passenger of the vehicle (18), wherein a voice of the user is further recognized by the context information system and a personal knowledge base stored in the storage unit is taken into account in responses for this user.

5. The method according to one of the preceding claims, wherein for each trip of the vehicle (18) activities of the context information system are stored as a self-contained knowledge base, so that responses to be given to inquiries are prepared and output in the context of this knowledge base.

6. The method according to one of the preceding claims, wherein the context information system, the vehicle sensor system, the user sensor system and the at least one storage unit and / or respective components of these systems are designed to be connected to the internet.

7. The method according to one of the preceding claims, wherein the component of a vehicle sensor system is selected from: front radar information device, rear radar information device, ultrasonic side detection device.

8. The method according to one of the preceding claims, wherein the component of a user sensor system is selected from: vehicle interior camera system, infrared sensor device, motion sensor device, 3D camera system.

9. A vehicle (18) that is designed to operate a method according to one of claims 1 to 8 before, after or during a trip.