Method and system for providing information requested in a motor vehicle about an object in the surroundings of the motor vehicle

Abstract

Method for providing information requested in a motor vehicle (1) about an object (15) in an environment of the motor vehicle (1), comprising the following steps: (a) detecting (S1) acoustic question data (18) describing a question from a user (2) of the motor vehicle (1) by means of a detection device (3), wherein the user (2) asks for at least one piece of information about an object (15) in an environment of the motor vehicle (1), (b) detecting (S2) sensor data (19) describing the user (2) of the motor vehicle (1) and / or the object (15) in the surroundings of the motor vehicle (1) by means of at least one sensor device (4, 5), (c) determining (S3) question content data (20) describing a content of the acquired question by applying a content determination criterion (21) to the acquired question data (18) and sensor data (19), (d) determining (S4) information data (27) which describe the requested at least one piece of information about the object (15) by performing a search query (26) on a search engine index (12) for the determined query content data (20) by means of an external computing device (11), (e) determining (S7) outputtable response data (30) describing an answer to the acquired question comprising the determined information by applying an answer determination criterion (29) to the determined information data (27), (f) Outputting (S8) the response data (30) in the motor vehicle (1) by means of an output device (6, 7), wherein position data (28) describing a position of the object (15) are determined (S5) and it is checked (S6) whether the determined position data (28) and information data (27) are assigned to the same object (15), wherein only if this is the case the response determination criterion (29) is applied.

Description

[0001] The invention relates to a method for providing information requested in the motor vehicle about an object in the surroundings of the motor vehicle. Furthermore, the invention relates to a system for implementing such a method.

[0002] A motor vehicle often has various sensor devices, for example a position detection device, a front camera, a rear camera, side cameras, a radar device, an infrared sensor and / or an ultrasonic sensor. The various sensor devices record sensor data that describe the surroundings of the motor vehicle, so that, for example, another vehicle approaching the motor vehicle and / or an obstacle for the motor vehicle can be detected. At least one sensor device that provides sensor data describing the interior of the motor vehicle, such as a driver observation camera, is often also arranged in an interior of the motor vehicle. Furthermore, the motor vehicle can have a voice assistant, by means of which a user of the motor vehicle can, for example, control a navigation system and / or an air conditioning system of the motor vehicle.

[0003] DE 10 2018 222 378 A1 discloses a device and a method for controlling the output of driver information to a driver of a vehicle. Information on sights or objects of general interest stored in a database is provided as driver information. Triggers for the output include, for example, a prolonged gaze and / or a gesture by the driver toward the sight or object of general interest.

[0004] US 2013 / 0030811 A1 shows sensors in the vehicle that monitor the driver's movements to determine the direction the driver is looking and gesture movements to determine what the driver might be pointing at. This information is combined with video camera data captured from the vehicle's external environment. The device uses this data to assist the speech dialogue processor in unambiguously identifying sentences uttered by the driver.

[0005] DE 10 2017 012 094 A1 shows a method for outputting travel information by means of a navigation arrangement and a navigation arrangement for carrying out the method, in which the travel information is generated and output to the user and / or the remote station on the basis of a voice communication carried out via a communication network between a user of the navigation arrangement and a remote station.

[0006] It is the object of the invention to provide a solution by means of which a comprehensive question and answer function is provided for a motor vehicle.

[0007] The problem is solved by the subject matter of the independent patent claims. Possible embodiments of the invention are specified in the dependent claims, the following description, and the figures.

[0008] The invention is based on the finding that the sensor data provided by sensor devices of the motor vehicle, which describe the surroundings of the motor vehicle, are often not correlated with the sensor data provided describing the interior of the motor vehicle. The sensor devices in the interior and exterior of the motor vehicle are thus typically decoupled from one another. Furthermore, the sensor data provided are often not correlated with the motor vehicle's voice assistant. This is because the voice assistant is often only designed to control a navigation system, a multimedia device, and / or an air conditioning system of the motor vehicle and is thus only suitable for supporting navigation and / or vehicle system control.

[0009] Furthermore, it is of interest to a user of the motor vehicle, particularly in a motor vehicle that is at least partially or even fully autonomous, to obtain additional information about an object located in the vicinity of the motor vehicle. In this case, it is particularly interesting to gain a better understanding of objects in the vicinity of the motor vehicle that, for example, cannot be recognized or classified by the motor vehicle itself. It is therefore sensible to obtain information about such an object for the user of the motor vehicle and to make it available to them, for example by means of an outsourced external computing device. Ultimately, the user of the motor vehicle should therefore be given the opportunity, for example, to ask an acoustic question about an object in the vicinity of the motor vehicle in the motor vehicle and to then receive a corresponding answer in the motor vehicle.Ultimately, a comprehensive and convenient question and answer function should be provided for the vehicle, for which the sensor devices and the voice assistant are related to each other.

[0010] A first aspect of the invention relates to a method for providing information requested in a motor vehicle about an object in the surroundings of the motor vehicle. In other words, the method is an information request method for a motor vehicle. The object is, for example, an object at the side of a road on which the motor vehicle is currently traveling. The object can be, for example, a poster, in particular an election poster.

[0011] In a first method step, for example, acoustic question data is recorded by means of a recording device. The recording device is preferably a microphone device, i.e. for example a microphone. The recording device is preferably a device of the motor vehicle. The question data describe a question from a user of the motor vehicle. The user is preferably a driver or another occupant of the motor vehicle. It is provided that the user asks for at least one piece of information about an object in the surroundings of the motor vehicle with the question. The surroundings of the motor vehicle are, for example, a field of vision of the user when the user looks out of a windshield, a side window or a rear window of the motor vehicle into the surroundings of the motor vehicle.Alternatively or additionally, the surroundings can be limited by the detection range of a sensor device of the motor vehicle for detecting the surroundings, so that the surroundings correspond to the detection range of the sensor device. The sensor device for detecting the surroundings is, for example, an external camera of the motor vehicle, for example, a front camera and / or a side camera.

[0012] The question about information about the object in the vicinity of the motor vehicle could, for example, be: “How old is the politician on the election poster up there?”. This is therefore a query about a detail regarding the object in the vicinity of the motor vehicle. In particular, the answer to the question does not have to be directly detectable by the motor vehicle’s sensor device. Furthermore, the answer preferably does not have to be determinable by the motor vehicle itself, i.e. a computing device of the motor vehicle. It can be a question that goes beyond the information presented by the object itself. An election poster, for example, typically contains a portrait of a politician and, where appropriate, written details about a party and / or the politician’s name. In contrast, an indication of the politician’s age is atypical for information on an election poster.The exemplary question about the politician's age on the election poster located in the vicinity of the vehicle thus goes significantly beyond the information presented by the object itself, i.e., the content displayed on the election poster. The recorded acoustic question data therefore preferentially does not describe a question about a local event and / or a point of interest, such as a landmark and / or an object of public interest. The question data preferentially describes a question about at least some detailed information about the object in the vicinity.

[0013] In a further method step, sensor data is recorded. This is done using a sensor device of the motor vehicle. The sensor device can be, for example, the front camera, the side camera, a rear camera, a radar device, a LIDAR device (LIDAR for light detection and ranging), an infrared sensor, an interior camera, a position detection device and / or an ultrasonic sensor. The sensor data describes the user of the motor vehicle. Alternatively or additionally, they describe the object in the surroundings of the motor vehicle. For example, data describing a viewing direction of the driver of the motor vehicle can be recorded as sensor data describing the user using the interior camera and made available as sensor data.For example, the front camera can be used to detect the object in the surroundings of the motor vehicle, so that static and / or moving image data of the object can be captured and provided in the form of camera data.

[0014] In a further method step, question content data is determined. The question content data describes the content of the recorded question. The question content data is determined by applying a content determination criterion to the recorded question data and sensor data. The content determination criterion preferably comprises a rule and / or an algorithm by means of which the question can be understood in terms of its content. Based on the recorded line of sight as sensor data, it can be determined, for example, which object in the surroundings of the motor vehicle may or may not be affected by the recorded question. For example, an object in the surroundings that was not previously in the user's line of sight is eliminated as a possible object to which the question refers.If, however, the user looks at the object in the environment, such as the election poster, for several seconds, it can be assumed that the user has actually seen this object, so that the object to which the question refers can be identified based on the sensor data. Preferably in combination with the user's gaze direction, i.e., the sensor data describing the user of the vehicle, and the sensor data describing the object in the vehicle's environment, detailed information about the object in the environment is available. This is then related to the question using the content determination criterion, by taking the question content data into account.

[0015] By determining the question content data, the user's intent is determined, and the corresponding object is classified and thus assigned. The content determination criteria for this can be based on speech recognition and object recognition methods, in particular object recognition in image data. Speech recognition is performed, for example, by a vehicle's voice assistant. Preferably, the content of the question is first recorded verbally, i.e., speech recognition is performed. This is followed, preferably, by assigning the linguistic content to the actual object in the environment, thus determining which object the question requests at least some information about.In the case of the election poster example and the question about the age of the politician on the election poster, the question content data preferably includes an image of the politician taken from the camera data and the question about the age of the person on the image.

[0016] A further method step comprises determining information data that describe the requested at least one piece of information about the object. The information data is determined by conducting a search query on a search engine index. The search query requests the determined query content data. The search query is carried out using an external computing device and thus not with the aid of the motor vehicle itself. In other words, the information data is determined by submitting a search query for the determined query content data to a search engine index. The search query is carried out using an external computing device. The search engine index is, for example, a database on which an internet search, in particular an image-based and / or keyword-based internet search, is based.The search engine index can, for example, be assigned to a commercial, a freely accessible, and / or a non-commercial internet search engine. When performing the process of determining the information data, the external computing device can further access at least one external data source, for example, by conducting an internet search. The external computing device is preferably a backend, for example, a computer, a server, and / or a cloud system.

[0017] In the example of the election poster and the question about the age of the politician on the election poster, the name of the politician is preferably first searched for using an image-based search query based on the politician's image, i.e. the question content data. Once the politician's name has been determined, a keyword-based search can be carried out, for example, for the age of the politician identified by his name. When carrying out the search query, an image search is therefore preferably first carried out based on the object's sensor data, here for example the captured image of the politician, in order to identify, classify and / or name the object. Subsequently, a further search, preferably more specific than the image search, can be carried out based on the question data and the question content data. In the example mentioned, the age of the identified politician is researched.The information data then ultimately shows the politician’s age.

[0018] Furthermore, one method step provides for the determination of outputtable response data. The response data describe an answer to the recorded question that encompasses the determined information. The response data is determined by applying an answer determination criterion to the determined information data. The answer determination criterion is a rule and / or an algorithm based on which, for example, at least one sentence is formulated that processes the information data as an answer understandable to the user. In the example mentioned, the response data can, for example, describe the sentence: "This politician is 43 years old." The response data is thus in an outputtable form.

[0019] It can be provided that the determined response data are transmitted to the motor vehicle, provided they were determined by means of the external computing device. In the motor vehicle itself, the response data are output by means of an output device. The output device is preferably a loudspeaker of the motor vehicle, i.e. a loudspeaker device for acoustically providing the response. Alternatively or additionally, the output device can be an optical display device for optically providing the response in the motor vehicle, i.e., for example, a display with a touch-sensitive display surface and / or a screen. The response is preferably output acoustically as a response sentence described by the response data, and preferably also optically.

[0020] Alternatively or additionally, the detection device and / or the output device can be comprised of a mobile device positioned in the motor vehicle. The mobile device is, for example, a mobile phone, such as a smartphone, or a tablet.

[0021] Data is preferably transmitted between the motor vehicle and the external computing device via a communication connection between corresponding communication interfaces in the motor vehicle and in the external computing device. The communication connection can be implemented as a wireless connection, for example via a wireless local area network (WLAN), a Bluetooth connection, and / or a mobile data network, for example based on the Long Term Evolution (LTE), Long Term Evolution Advanced (LTE-A), Fifth Generation (5G), or Sixth Generation (6G) mobile communications standards.

[0022] The described method achieves a particularly advantageous interaction between the individual sensor devices and the motor vehicle's voice assistant, enabling the answer to be provided to a question posed by a motor vehicle user by drawing on the knowledge of a search engine index and thus, for example, from websites on the Internet. Ultimately, this provides a simple mechanism for obtaining desired information about an object in the vehicle's surroundings. This is based on sensor data from the motor vehicle, primarily camera data. By outsourcing the actual determination of the answer to the question to the external computing device using the search query, resources, for example computing resources, are saved within the motor vehicle and, at the same time, a wide range of benefits are created for customers and passengers.Overall, a particularly convenient question and answer function is provided for a motor vehicle. Overall, dynamic and convenient information retrieval about the object in the vehicle's surroundings is provided.

[0023] The invention provides that position data describing a position of the object is determined. This is preferably done by evaluating the position data of the motor vehicle, which is detected by the position detection device, and distance data that describes the distance of the object from the motor vehicle. The position data are preferably coordinates of the object. A check is also carried out to determine whether the determined position data and the determined information data are assigned to the same object. Only if this is the case, i.e. only if the determined position data and information data are assigned to the same object, i.e. to an identical object, is the response determination criterion applied to the information data. Thus, after the information data has been determined, a check is first carried out to determine whether this actually relates to the object about which the user wishes information.

[0024] It may, for example, be the case that in response to the question "How high is that mountain over there?" the height of several mountains in the vicinity of the motor vehicle is determined as part of the determination of the information data. It can then be determined whether the mountain to which the user pointed with one hand, for example, and whose position data was thus determined, is actually the mountain to which the determined information data refers. If, for example, the information data contains further height information for visually similar mountains, but these are not located in the vicinity of the motor vehicle or therefore do not match the position data, it can be concluded that this is not the desired information data. The determination of response data for such unwanted information data is then prevented. Ultimately, the user's intent is therefore checked.This allows the actual desired answer to be identified and selected, for example, when different information data are available. This is particularly useful when multiple search results are generated for the search query, so that the information data contains information about multiple objects and / or multiple pieces of information about similar objects. Using the described check based on the position data, the information data actually requested by the user can then be selected. This ensures that the output response data always describes the actual desired information data and thus the actual answer to the user's question.

[0025] The invention also includes embodiments which provide additional advantages.

[0026] An advantageous embodiment provides that the object in the environment is at least one of the following objects: a poster, in particular an advertising poster and / or an election poster; an electronic display, in particular an electronic advertising display; a living being, in particular a plant and / or an animal; and / or a landscape object. In other words, information about, for example, print media, such as the poster, can be requested. Alternatively or additionally, information about advertising, in particular digital advertising on the electronic display, such as an electronic display screen or display, can be requested. For example, a query can be made about a product shown on the poster and / or the electronic display.For example, you can ask where such a product can be purchased, whether it is currently available and / or where the nearest store is where the item shown on the poster can be purchased.

[0027] Alternatively or additionally, information on flora and / or fauna can be requested. With regard to living beings, the question can, for example, be directed at a type of tree, a type of flower and / or a type of plant. For example, the user can ask what type of tree they are currently driving past in their motor vehicle. Analogously, they can, for example, ask about an animal species or breed, such as a dog breed, that the motor vehicle is currently driving past. The landscape object can, for example, be a mountain range, a body of water and / or a valley that the motor vehicle is currently driving past.The question directed at the landscape object can, for example, request a name of the landscape object and / or a height, width, length and / or depth of the landscape object, for example, request the height of a mountain and / or the maximum depth of a lake and / or a width of a valley and / or the length of a river from a source to a mouth.

[0028] In particular, it may be provided that the question does not refer to a point of interest, i.e., for example, a tourist attraction and / or an object of general interest. The reason for this may be that information about the point of interest is typically already stored in the vehicle's navigation system, and thus the complex search query to the search engine index is not required to display details about the point of interest in the vehicle. Furthermore, it may be ruled out that the object is a commercial point of interest, such as a gas station, a restaurant, and / or a shop that the vehicle passes by, since detailed information about such points of interest is often stored in the vehicle's navigation system.

[0029] It is preferably provided that the question data does not describe a question that merely asks about the object itself, for example the question: "What is that?". Rather, the invention is directed at obtaining at least one piece of additional information about the object that can only be obtained from the object by submitting a corresponding search query to the search engine index. Using the types of objects mentioned, numerous different cases are mentioned in which a user could become curious about the object while driving past and ask a question about the object. The objects mentioned are not to be understood as a complete list. Further or alternative objects in the vicinity of the motor vehicle are possible.

[0030] An additional embodiment provides that the acquired sensor data describe at least one of the following information: a head direction of the user, a gaze direction of the user, a gesture of the user, a position of the motor vehicle, an orientation of the motor vehicle, a distance of the motor vehicle from the object and / or static or moving images of the object.

[0031] The user's head direction, gaze direction, and / or gesture can preferably be detected using the interior camera of the motor vehicle. An evaluation device can be provided in the motor vehicle and / or in the external computing device, by means of which camera data from the interior camera is evaluated in such a way that the user's head direction, gaze direction, and / or gesture can be recognized and, in particular, identified. A rule required for this evaluation can be integrated into the content determination criterion.

[0032] The position of the motor vehicle is preferably determined using the motor vehicle's position detection device. A navigation system can be used for position determination. Position determination is preferably carried out using a global navigation satellite system (GNSS), in particular the global positioning system (GPS). The orientation of the motor vehicle is, for example, a north-south orientation and / or an orientation relative to the object in the vehicle's surroundings.Based on the orientation of the motor vehicle to the object, it can be determined, for example, which object of several possible objects is most likely meant by the user of the motor vehicle in the context of the question, since, for example, the orientation of the motor vehicle is directed towards this object and the object is not located in a blind spot of the surroundings of the motor vehicle that is not visible to the user, i.e. a blind spot.

[0033] The distance to the object can, for example, be used to identify the object more precisely. For example, when asking about the height of the mountain in the vicinity of the motor vehicle, the distance to the object can be used to determine which mountain the question refers to. The object can therefore be described in more detail using distance data, which is recorded using a radar, lidar device and / or infrared sensor, for example, so that the question content data can include a name and / or position of the mountain to which the question refers. For this purpose, the recorded distance between the motor vehicle and the object can be related to a map stored in the motor vehicle and thus to map data from the motor vehicle's navigation system.This would allow the mountain to be identified based on the map data provided by the navigation system, so that, for example, an image search based on camera data from an external camera of the vehicle is not necessary in order to carry out a search for the height of the mountain.

[0034] The static and / or moving images of the object are preferably the camera data, which can be used, for example, for object identification and / or object recognition when applying the content determination criterion. Ultimately, therefore, a wealth of information is available from which the question content data can be reliably determined.

[0035] In a further embodiment, the content determination criterion comprises a search term recognition criterion. Search term data is determined by applying the search term recognition criterion to the acquired question data. The search term data describes a search term for the search query. The question content data comprises the search term data. The method step for determining the question content data can therefore be divided into a first sub-step. In other words, this comprises speech recognition and thus understanding the user's intent, i.e., the query for information about the object. The search term recognition criterion is preferably based on speech recognition or speech content recognition methods. The search term recognition criterion is an algorithm and / or a rule by means of which the search term data can be determined, i.e., is determined.Alternatively or additionally, the search term recognition criterion can also be applied to the sensor data. The search term data is preferably in the form of terms, i.e., words that can be searched for in the search engine index. In the above example, the search term data includes, for example, the terms "age" and "politician." This allows the content of the question to be recognized with particular precision.

[0036] Furthermore, one embodiment provides for the content determination criterion to comprise an object criterion. By applying the object criterion to the acquired sensor data, object data is determined. The object data describes the object to which the acquired question relates. The question content data comprises the object data. Thus, with the aid of the object data, the object in the surroundings of the motor vehicle about which information is desired is detected and preferably identified. The object criterion comprises a rule and / or an algorithm, the application of which can determine the object data. The object criterion is preferably based on image processing and / or data processing methods. The object data can be, for example, image data and / or other sensor data.The object data is preferably already processed image data, so that, for example, a background and / or at least one other object has already been removed from the sensor data. The object data is thus cleaned data. The object data preferably describes only the object to which the question refers. In the case of the above example, for example, the object data only includes the image of the politician as it appears on the election poster, but preferably without any other poster content, such as a background and / or text.

[0037] Ultimately, the object detection criterion can reliably provide a content detection criterion, the application of which can actually provide the content of the captured questions in the form of question content data.

[0038] Furthermore, one embodiment provides that the content determination criterion is based on machine learning. The content determination criterion can, for example, comprise multi-layer learning methods, such as deep learning, and / or a programming framework for data stream-oriented programming, such as TensorFlow. Preferably, the content determination criterion is created based on an artificial neural network, wherein a training phase of the artificial neural network is preferably already completed when the described method is carried out. In particular, for typical use cases, such as questions about living beings, landscape objects, posters, and / or electronic displays, the content determination criterion can already be trained, for example, using a database with corresponding image data of such objects.The content determination criterion can therefore quickly and reliably determine and provide the question content data.

[0039] According to a particularly preferred embodiment, it is provided that the acquired question data and / or sensor data are transmitted to the external computing device. The external computing device then carries out, for example, the method steps for determining the question content data, the information data, and / or the outputtable response data. Alternatively, it can be provided that the question content data and the information data are determined using the external computing device, and the determined information data are provided to the motor vehicle for determining the response data. At least the acquisition of the acoustic question data and the sensor data, as well as the output of the response data in the motor vehicle, is preferably carried out using the motor vehicle itself.

[0040] The output of the response data and the acquisition of the acoustic question data can be performed alternatively or additionally using a terminal device located in the motor vehicle. The terminal device can be, for example, a mobile phone, such as a smartphone, and / or a tablet, and / or a computer, such as a laptop. Ultimately, this can save computing resources within the motor vehicle, as the complex process of determining the information data and question content data is outsourced to the external computing device.

[0041] It can further be provided that only after the search term data have been determined are sensor data from the motor vehicle specifically requested by the external computing device, so that the sensor data from the motor vehicle are transmitted to the external computing device retroactively for the time at which the acoustic question data were recorded, so that the latter can fully determine the question content data, i.e. the object data can also be determined.

[0042] In addition, one embodiment provides that the response data describe an acoustically outputtable response and an optically outputtable response. Ultimately, the response data therefore comprises two different partial response data, namely acoustically outputtable partial response data and optically outputtable partial response data. The acoustically outputtable response and the optically outputtable response preferably differ in at least one piece of the determined information. For example, the acoustically outputtable response can be a response sentence to the user's question. In the above example, this response sentence is, for example, "The politician is 43 years old." The response sentence is output, for example, via the loudspeaker in the motor vehicle. The optically outputtable response is preferably output on a screen, i.e., a display device in the motor vehicle, and can include additional details.For example, the visually outputtable answer can also include the politician's name, their birthday, and / or provide a link to additional information about the politician. The link can, for example, point to an entry in an online encyclopedia, an online article by a party to which the politician belongs, and / or the politician's personal website. This results in a particularly satisfactory compilation of response data for a voice and text output of the answer to the recorded question about the object. Furthermore, additional information can be provided, which can further increase the convenience of the question and answer function.

[0043] A further aspect of the invention relates to a system for providing information requested in a motor vehicle about an object in the surroundings of a motor vehicle. The system comprises the motor vehicle and an external computing device. The system is designed to carry out the method described above. The advantageous embodiments described in connection with the method according to the invention, as well as combinations of the aforementioned embodiments, apply analogously, where applicable, to the system according to the invention.

[0044] A further aspect of the invention relates to a motor vehicle that can detect at least the acoustic query data and / or sensor data and / or output the response data. Furthermore, the motor vehicle can receive data from the external computing device. It can further be provided that the motor vehicle can also determine the query content data and the response data. It can further be provided that the motor vehicle itself carries out the search query to the search engine index for determining the information data and can thus access external data sources via a communications connection. The motor vehicle according to the invention is preferably designed as a motor vehicle, in particular as a passenger car or truck, or as a passenger bus or motorcycle.

[0045] The invention also includes the external computing device and / or an evaluation device for the motor vehicle. The external computing device and / or the evaluation device can have a data processing apparatus or a processor device that is configured to carry out an embodiment of the method according to the invention. For this purpose, the processor device can have at least one microprocessor and / or at least one microcontroller and / or at least one FPGA (Field Programmable Gate Array) and / or at least one DSP (Digital Signal Processor). Furthermore, the processor device can have program code that is configured to carry out the embodiment of the method according to the invention when executed by the processor device. The program code can be stored in a data memory of the processor device.

[0046] The invention also encompasses combinations of the features of the described embodiments. The invention therefore also encompasses implementations that each comprise a combination of the features of several of the described embodiments, unless the embodiments are described as mutually exclusive.

[0047] Exemplary embodiments of the invention are described below. Shown are: Fig. 1 a schematic representation of a system for providing information requested in the motor vehicle about an object in the surroundings of the motor vehicle; and Fig. 2 shows a schematic representation of a signal flow graph of a method for providing information requested in a motor vehicle about an object in the environment of the motor vehicle.

[0048] The exemplary embodiments explained below are preferred embodiments of the invention. In the exemplary embodiments, the described components of the embodiments each represent individual features of the invention that can be considered independently of one another, each of which also develops the invention independently of one another. Therefore, the disclosure is intended to encompass combinations of the features of the embodiments other than those shown. Furthermore, the described embodiments can also be supplemented by further features of the invention already described.

[0049] In the figures, the same reference symbols designate elements with the same function.

[0050] Fig. 1 shows a motor vehicle 1. A user 2 is located in the motor vehicle 1. The user 2 is a passenger of the motor vehicle 1, in particular a driver of the motor vehicle 1.

[0051] The motor vehicle 1 has a detection device 3, which here is embodied as a microphone of the motor vehicle 1, by way of example. The motor vehicle 1 has at least one sensor device 4, 5. In this example, the sensor device 4 is an interior camera of the motor vehicle 1, and the sensor device 5 is a front camera of the motor vehicle 1. By means of the interior camera as sensor device 4 of the motor vehicle 1, an interior of the motor vehicle 1 can be detected, that is to say, for example, a head direction, a line of sight, and / or a gesture of the user 2. By means of the front camera as sensor device 5, an environment of the motor vehicle 1 can be detected. The environment of the motor vehicle 1 can be limited by a detection range of the sensor device 5 and / or by a field of vision of the user 2.

[0052] Motor vehicle 1 has, for example, two output devices 6, 7. Output device 6 is, for example, a loudspeaker for outputting acoustic information in motor vehicle 1. Output device 7 is, for example, designed as a display device, i.e., for example, as a touch-sensitive or non-touch-sensitive screen or display. A visually displayed message and / or a visual indication can be output in motor vehicle 1 by means of the display device. The display device as output device 7 is preferably arranged in the region of a center console of motor vehicle 1.

[0053] The motor vehicle 1 has a position detection device 8. This can, for example, be integrated into a navigation system of the motor vehicle 1. The position detection device 8 can be used to determine the current position of the motor vehicle 1. The motor vehicle 1 also has a communications interface 9. The motor vehicle 1 further has a control device 10. The control device 10 comprises, for example, a processor. The control device 10 can at least control all method steps performed in the motor vehicle 1.

[0054] In Fig. 1, an external computing device 11 is shown. This also has the communication interface 9. A communication connection can be established between the communication interface 9 of the motor vehicle 1 and the communication interface 9 of the external computing device 11. This is preferably designed as a wireless communication connection. Fig. 1, the wireless communication link is outlined with a double-headed dashed line.

[0055] The external computing device 11 can have a search engine index 12, i.e., it can have an online search function. For this purpose, it can establish and maintain a communication connection to at least one external data source 13. Finally, the external computing device 11 is configured to perform an internet search and / or internet research based on the search engine index 12.

[0056] Motor vehicle 1 is currently traveling on a road 14. An object 15 is located at the edge of road 14. Object 15 is exemplified here as a poster, specifically an election poster. A politician 16 is depicted on the poster as object 15. In the example cited here, user 2 points at object 15. This gesture can be detected by the interior camera as sensor device 4.

[0057] Furthermore, in Fig. 1 outlines a system 17. The system 17 comprises the motor vehicle 1 and at least the external computing device 11. The system 17 can also include the external data sources 13.

[0058] Fig. 2 shows a schematic representation of a method for providing requested information in the motor vehicle 1. The requested information is provided in the motor vehicle 1 and relates to the object 15 in the environment of the motor vehicle 1.

[0059] In a first method step S1, acoustic question data 18 is captured by the capture device 3. The question data 18 describes a question from user 2. With this question, user 2 asks for at least one piece of information about the object 15 in the surroundings of the motor vehicle 1. In the example outlined here, user 2 asks the following question, for example: "How old is the politician on the poster?" They also point their hand in the direction of object 15, i.e., in this case, the election poster. Furthermore, user 2 is looking toward object 15. The question data 18 is captured acoustically, i.e., user 2 makes a voice input into the microphone as capture device 3.

[0060] In a further method step S2, sensor data 19 is acquired by at least one of the sensor devices 4, 5. The sensor data 19 describe the user 2 and the object 15 in the surroundings of the motor vehicle 1. In this example, the sensor data 19 describing the user 2 describes the head and gaze direction of the user 2 as well as the pointing gesture of the user 2 towards the object 15 in the form of camera data from the interior camera. The sensor data 19 describing the object 15 is, for example, camera data from the front camera as the sensor device 5. Alternatively or in addition to such static or moving images or image data of the object 15, the sensor device 4, 5 can determine a distance between the motor vehicle 1 and the object 15, for example, using an infrared sensor, a radar device, and / or a lidar device of the motor vehicle 1.Furthermore, the sensor data 19 can include a position and / or orientation of the motor vehicle 1. These are preferably determined by the position detection device 8 of the motor vehicle 1. Finally, a state in the interior and exterior of the motor vehicle 1 is detected by the sensor device 4, 5.

[0061] In a method step S3, question content data 20 is determined. Method step S3 can be performed after method step S2 or directly after method step S1. The question content data 20 describes the content of the acquired question. The question content data 20 is determined by applying a content determination criterion 21 to the acquired question data 18 and sensor data 19. In detail, the content determination criterion 21 preferably comprises two subcriteria. These are a search term recognition criterion 22 and an object criterion 23.

[0062] By applying the search term recognition criterion 22 to the acquired question data 18, search term data 24 can be determined, which is comprised of the question content data 20. The search term data 24 describes a search term for a search query. In this example, the search term recognition criterion 22 performs, for example, speech recognition so that a user request, i.e., the question of user 2, can be understood using the determined search term data 24. In the example mentioned here, the search term data 24 describes, for example, the two search terms "age" and "politician" 16. Ultimately, the search term data 24 encompasses the user intent of user 2, i.e., it describes what information they would like to receive.

[0063] When the object criterion 23 is applied to the recorded sensor data 19, object data 25 is determined. The object data 25 is also included in the question content data 20. The object data 25 describes the object 15 to which the recorded question refers. In the example mentioned here, the object data 25 is, for example, image data showing the politician 16. For this purpose, the camera data of the front camera as sensor device 5 is evaluated in such a way that, taking into account the recorded sensor data 19 describing the user 2, it can be deduced that the election poster is the object 15 to which the question of user 2 refers. In addition, image data is extracted from the total data recorded, for example, by the front camera, which only describes, for example, the portrait of the politician 16 shown on the election poster.For example, a background of object 15 and / or other objects 15 in the vicinity of object 15, here the election poster, are removed from the sensor data 19, so that the object data 25 is only directed at object 15. The total recorded question content data 20 thus includes the specific question of user 2 as well as the classified object 15.

[0064] The content determination criterion 21 is preferably based on machine learning methods, in particular on an artificial neural network. The content determination criterion 21 preferably comprises a rule and / or an algorithm based on multi-layer learning, such as deep learning, and / or a programming framework for data stream-oriented programming, such as TensorFlow.

[0065] In a method step S4, a search query 26 is performed for the determined question content data 20. The search query 26 is directed to the search engine index 12 and is preferably performed using the external computing device 11. Within the scope of the search query 26, information data 27 are determined that describe the requested at least one piece of information about the object 15. In detail, for example, the object data 25 are used as the basis for an image search based on the search engine index 12, so that, for example, the politician 16 depicted as object 15 on the election poster can be identified. Based on the search term data 24 of the question content data 20, a search for the age of the now identified politician 16 can also be performed. In this example, the information data 27 thus preferably includes an age indication of the politician 16.

[0066] The external computing device 11 can access the external data source 13 to execute the search query 26. Preferably, the determination of the query content data 20, i.e., method step S3, is already performed by the external computing device 11. Alternatively or additionally, it is possible for method steps S1, S2, and S3 to all be performed in the motor vehicle 1. At least method steps S1 and S2 are always performed by the motor vehicle 1 and / or at least partially by a mobile device positioned in the motor vehicle 1. The mobile device can be a mobile phone, such as a smartphone, and / or a tablet of the user 2, which is arranged in the motor vehicle 1.

[0067] In a method step S5, position data 28 describing a position of object 15 can be determined. In method step S6, a check is then performed to determine whether the determined position data 28 and the information data 27 determined in method step S4 are assigned to the same object 15. The position data 28 used here are determined based on the vehicle position data of motor vehicle 1 determined by means of the position detection device 8 of motor vehicle 1 in combination with the determined distance between motor vehicle 1 and object 15. The orientation of motor vehicle 1 can also be taken into account.

[0068] If the check in method step S6 is negative, it can be concluded that the position data 28 does not match the information data 27, so that the information data 27 cannot be an answer to the question acquired using the question data 18. The search query 26 can then be performed again, for example. If the determined position data 28 and the determined information data 27 are assigned to the same object 15, i.e., if they match each other, the method continues with a step S7.

[0069] If, for example, the question of user 2 is directed at a geographically fixed object 15, such as a landscape object such as a mountain near road 14, a check can be carried out in method step S6 to determine whether, for example, the information data 27 determined can actually be assigned to the mountain that is located as object 15 in the surroundings of motor vehicle 1. If, for example, information about a mountain is obtained in the search query 26 that is clearly outside the field of vision of user 2, the search results found in error in this way can no longer be taken into account. This ensures that, for locally fixed objects 15, a check of the actual connection between the information data 27 and the object 15 that is assigned to the position data 28 is carried out.

[0070] In method step S7, an answer determination criterion 29 is applied to the determined information data 27. Outputtable answer data 30 is determined, which describes an answer to the recorded question that encompasses the determined information. In the example mentioned, the answer data 30 includes, for example, the information that the politician is 16 years old.

[0071] In a further method step S8, the response data 30 is output in the motor vehicle 1, specifically by means of the output device 6, 7. Here, a distinction is made between two partial response data 31, namely first partial response data 31, which describes an acoustically outputtable response, and second partial response data 31, which describes an optically outputtable response. The two partial response data 31 preferably differ from one another in at least one piece of information. For example, the first partial response data 31 can be displayed by means of the loudspeaker as output device 6, and the second partial response data 31 can be displayed by means of the display device as output device 7. For example, the optically outputtable response can contain more details than the acoustically outputtable response. The acoustically outputtable response in the above example is preferably the sentence: "This politician is 43 years old."

[0072] The visually outputtable answer may also include details about the politician 16 himself, such as his date of birth, his name, his party affiliation and / or a link to a website about the politician 16.

[0073] Method steps S6 and S7 are preferably performed by the external computing device 11. At least method step S8 is performed by and / or in the motor vehicle 1.

[0074] In addition or alternatively to the above example, the object 15 can be an advertising poster, an electronic display, in particular an advertisement, a living being, in particular a plant and / or an animal, and / or a landscape object, such as a mountain.

[0075] Overall, the examples demonstrate a method for dynamically obtaining information from objects 15 in the environment of motor vehicle 1. A multimodal question and answer function is performed using a voice assistant, with user 2 obtaining information about object 15 in the environment. Question data 18 is transmitted to a backend system, preferably external computing device 11, controlled, for example, by control device 10 of motor vehicle 1. Control device 10 may be referred to as a session handler. An object classifier, i.e., object criterion 23, is operated by control device 10 using the user intent, which is present in the form of search term data 24, and the relevant image sections, i.e., sensor data 19 of object 15.Based on this, an image-processing object classification is initiated, including an optional web request, to return the classified object 15. Thus, process steps S3 and S4 are performed. Subsequently, the user intentions, including the located classified object 15, are prepared for programming interface requests (Application Programming Interface (API) calls), and the desired information, i.e., the information data 27, is determined. The API calls themselves include all interfaces from third-party providers and service providers connected to the system 17, which are identified and operated based on the use case.

[0076] The third-party providers and service providers are, for example, the external data source 13. If a direct assignment has not yet been implemented at the time of the query, i.e. at the time the question data is collected 18, a standard online search is used, which, due to the larger solution space, entails greater effort and follow-up.

[0077] After receiving the data from the external data sources 13, a filtering process is performed to determine the specific customer intent, i.e., the question of user 2. For this purpose, the received object data 25 is homogenized based on the condensed user intent, i.e., the search term data 24. Furthermore, the input and output mapping is analyzed and optimized. Subsequently, in process steps S7 and S8, the data for the voice and text output of the digital assistant is compiled. The digital assistant here is therefore the microphone in combination with the loudspeaker of motor vehicle 1.

Claims

[1] Method for providing information requested in a motor vehicle (1) about an object (15) in an environment of the motor vehicle (1), comprising the following steps: (a) detecting (S1) acoustic question data (18) describing a question from a user (2) of the motor vehicle (1) by means of a detection device (3), wherein the user (2) asks for at least one piece of information about an object (15) in an environment of the motor vehicle (1), (b) detecting (S2) sensor data (19) describing the user (2) of the motor vehicle (1) and / or the object (15) in the surroundings of the motor vehicle (1) by means of at least one sensor device (4, 5), (c) determining (S3) question content data (20) describing a content of the acquired question by applying a content determination criterion (21) to the acquired question data (18) and sensor data (19), (d) determining (S4) information data (27) which describe the requested at least one piece of information about the object (15) by performing a search query (26) on a search engine index (12) for the determined query content data (20) by means of an external computing device (11), (e) determining (S7) outputtable response data (30) describing an answer to the acquired question comprising the determined information by applying an answer determination criterion (29) to the determined information data (27), (f) Outputting (S8) the response data (30) in the motor vehicle (1) by means of an output device (6, 7), wherein position data (28) describing a position of the object (15) are determined (S5) and it is checked (S6) whether the determined position data (28) and information data (27) are assigned to the same object (15), wherein only if this is the case the response determination criterion (29) is applied. [2] Method according to claim 1, wherein the object (15) in the environment is at least one of the following objects (15): - a poster, in particular an advertising poster and / or an election poster, - a digital display, in particular an advertisement, - a living being, in particular a plant and / or an animal, and / or - a landscape object (15). [3] Method according to one of the preceding claims, wherein the acquired sensor data (19) describe at least one of the following information: - a head direction of the user (2), - a viewing direction of the user (2), - a gesture of the user (2), - a position of the motor vehicle (1), - an alignment of the motor vehicle (1), - a distance of the motor vehicle (1) to the object (15), and / or - static and / or moving images of the object (15). [4] Method according to one of the preceding claims, wherein the content determination criterion (21) comprises a search term recognition criterion (22) and by applying the search term recognition criterion (22) to the acquired question data (18), search term data (24) are determined which describe a search term for the search query (26), wherein the question content data (20) comprises the search term data (24). [5] Method according to one of the preceding claims, wherein the content determination criterion (21) comprises an object criterion (23) and by applying the object criterion (23) to the acquired sensor data (19), object data (25) are determined which describe the object (15) to which the acquired question relates, wherein the question content data (20) comprises the object data (25). [6] Method according to one of the preceding claims, wherein the content determination criterion (21) is based on machine learning. [7] Method according to one of the preceding claims, wherein the acquired question data (18) and / or sensor data (19) are transmitted to the external computing device (11) and the external computing device (11) carries out the method steps (c) to (e). [8] Method according to one of the preceding claims, wherein the response data (30) describe an acoustically outputtable response and an optically outputtable response, which differ in particular in at least one piece of information of the determined information. [9] System (17) for providing information requested in a motor vehicle (1) about an object (15) in an environment of the motor vehicle (1), wherein the system (17) comprises the motor vehicle (1) and an external computing device (11) and is designed to carry out a method according to one of the preceding claims.

Images