Method for providing a speech dialogue in sign language in a speech dialogue system of a vehicle

Abstract

A method for providing a speech dialogue in sign language in a speech dialogue system of a vehicle (1), wherein the following steps are performed: - performing an optical detection of input information of a vehicle occupant (2); - performing an evaluation of the detected input information; - providing a visual output (131) in sign language as a function of the evaluation.

Description

Technical Field

[0001] The invention relates to a method for providing a speech dialogue in sign language in a speech dialogue system of a vehicle. The invention also relates to a system and a vehicle. Background Art

[0002] A verbal dialogue system in a vehicle enables the operation of the vehicle and the exchange of information with the vehicle by verbal communication. In the age of automated driving functions, an additional driving experience can be provided in this way, which consists in being able to communicate with the vehicle for information exchange and / or for entertainment. Gesture control of instruments in a vehicle is also known. However, communicating with the vehicle remains problematic for people with speech impairments or people who are deaf and hard of hearing.

[0003] US 2005 / 0271279 A1 discloses interaction with a machine by means of gestures in order to provide commands to the machine.

[0004] US 2013 / 0261871 A1 also discloses a gesture-based control.

[0005] US2016 / 0107654A1 discloses an activation method for user input. Summary of the invention

[0006] The object of the present invention is therefore to at least partially eliminate the disadvantages described above. In particular, the object of the present invention is to provide a solution for a silent language dialogue system.

[0007] The above-mentioned object is achieved by a method for providing a language dialogue in sign language in a language dialogue system of a vehicle, a system for providing a language dialogue in sign language for a vehicle, and by a corresponding vehicle. Further features and details of the invention are obtained from the corresponding description and the drawings. Features and details described in connection with the method according to the invention naturally also apply in connection with the system according to the invention and the vehicle according to the invention, and vice versa, respectively, so that reference can always be made to each other with respect to the disclosure of the individual inventive aspects.

[0008] The object is achieved in particular by a method for providing a speech dialogue in a sign language in a speech dialogue system, in particular a silent speech dialogue system, of a vehicle. In this case, provision is made in particular for the following steps to be carried out, preferably successively in the order specified or in any order, wherein individual steps may also be carried out repeatedly if necessary:

[0009] - optical detection of input information from a vehicle occupant, in particular by means of an image detection device;

[0010] - performing an evaluation of the detected input information, in particular by means of a processing device;

[0011] - providing a visual output in sign language as a function of the evaluation, in particular via an output device.

[0012] In this way, a silent execution of a verbal dialogue can be provided by means of the verbal dialogue system, so that a natural language dialogue can be carried out with the vehicle. The natural language dialogue is formed in particular from input information (e.g. in the form of "incoming words" or questions or responses or commands) and outputs associated therewith (e.g. in the form of "outgoing words" or responses to questions or questions or feedback to commands). In this case, a particular advantage is that the output in sign language is provided so that the dialogue can also be carried out for people who have difficulty in carrying out the dialogue using spoken language. Thus, it can be possible that the dialogue is completely free of spoken language.

[0013] It is also possible that the input information has information about the sign language of the vehicle occupant. This allows the dialogue provided by the language dialogue system to be provided completely in sign language. For this purpose, the input information includes in particular information about gestures and / or facial expressions and / or unspoken words and / or body postures of the vehicle occupant, which can be analyzed by an evaluation of the processing device. In order to detect the input information with this information, for example, an image recording of the vehicle occupant can be performed by an image acquisition device. The evaluation can also include image analysis and / or machine learning methods in order to, for example, perform a classification of the input information with respect to predefined gestures and / or symbols. In this way, the situation in the vehicle interior can be analyzed by the language dialogue system and used in the dialogue.

[0014] Furthermore, it is optionally provided that the execution of the evaluation comprises the following steps:

[0015] - performing a detection of an interaction of a vehicle occupant in sign language with the language dialog system based on input information, wherein the input information is preferably embodied as an image recording of the vehicle occupant in order to detect the interaction at least as a gesture of the vehicle occupant based on the image recording;

[0016] - in particular only if the detection of the interaction is successful, performing an analysis of the input information, in particular an image analysis, for translating the sign language into content-related translation information, preferably text information, which can be evaluated by the language dialogue system;

[0017] - optionally: performing a detection of a further gesture of the vehicle occupant as a function of the input information, the further gesture being specific to the additional information;

[0018] - generating a response as a function of the translation information and preferably as a function of the additional information, in order to output the response in sign language via a visual output.

[0019] This allows a reliable evaluation of the input information in order to provide a dialog. The generation of the response can also include, for example, a content-based analysis of the translation information in order to determine, for example, content-based information as the response that is content-related to the input information (e.g., as a question). The output can be particularly advantageously realized in the form of an output in the form of a graphical avatar and / or abstract form and / or by means of symbols (i.e., based on symbols) and, if necessary, additionally as an audio output.

[0020] It may be possible that at least one gesture of a vehicle occupant can be detected when evaluating the input information. This gesture of the vehicle occupant can be detected as a gesture within the framework of a sign language and can be used accordingly to translate the sign language into a translation information. The gesture can also be detected as a further gesture that is specific to the additional information. A further gesture is, for example, pointing to an object, which correspondingly provides the selected object as additional information. Depending on the detected further gesture, in addition to the translation, this additional information can be used in the evaluation in order to generate the response. For example, the translation information indicates that there is a specific question related to the object. In other words, the translation information can represent a function and the additional information can represent parameters of the function. The response can then be generated as an output of the function.

[0021] The further gesture and the additional information are determined, for example, by an image capture device, in particular a first image capture unit such as a camera, capturing input information from which the further gesture (e.g. pointing to an object) can be derived. The additional information (i.e. the specific object) can be determined, for example, by a survey (further detection) of at least one other image capture device and / or at least one other image capture unit of the vehicle, by which the surroundings of the vehicle are detected and thereby the object is detected. For this purpose, for example, direction information (in which direction is it pointing?) is derived from the further gesture and is compared with the further detection of the object. It is also conceivable that the object is identified based on the direction information by evaluating location information about the current location of the vehicle and in particular comparing this location information with surroundings information and direction information.

[0022] According to an advantageous development of the invention, it can be provided that the response includes at least one of the following information (having the following content):

[0023] - object information about an object in the surroundings of the vehicle, wherein the additional information preferably provides for the selection of an object, preferably as a direction indication with respect to the object;

[0024] - weather information, in particular weather information about the weather at the location of the vehicle;

[0025] - Surroundings information, preferably information about the current location of the vehicle;

[0026] - news information, in particular news information about local news at the location of the vehicle;

[0027] - media information, such as videos or literary works,

[0028] In order to generate the response, the information is particularly preferably filtered with respect to the availability of the output of this information in sign language. This can provide a wide range of information channels for the language dialogue. In order to select an object, a further gesture according to the additional information is constructed, for example, as a finger gesture, etc., in which the object can be pointed to by the gesture. Here, according to this filtering, for example, only such content that is suitable for output in sign language can be considered. It is also possible that the information in text form is translated into sign language by the method and / or system according to the invention, for example, by means of a processing device. For this purpose, the processing device can also communicate with an external data processing device if necessary, so as to perform the translation, for example, via an external server or via cloud computing.

[0029] The subject of the invention is also a system for a vehicle for providing a verbal dialogue in sign language. The system can be implemented, for example, as a vehicle electronic device and is thus suitable for being fixedly integrated into the vehicle. In this case, it is provided that the system has at least one of the following components:

[0030] an image capture device, in particular a camera system, for capturing input information of a vehicle occupant, wherein the input information is preferably implemented as image information, in particular as an image recording;

[0031] a (especially electronic) processing device for evaluating the detected input information, wherein the processing device is embodied, for example, as a controller and / or an electronic device and / or a microcontroller and / or the like;

[0032] - Output means for producing a visual output in sign language based on the evaluation.

[0033] The system according to the invention thereby brings the same advantages as have already been described in detail with respect to the method according to the invention. In addition, the system can be suitable for carrying out the method according to the invention. It is also conceivable that the system is configured as a language dialogue system according to the method according to the invention in order to carry out the steps of the method.

[0034] The system can be designed, for example, as a technical component in a vehicle, which monitors the interior of the vehicle by means of an image detection device, in particular in the form of a camera system, and is able to recognize when a vehicle occupant (passenger) interacts with the system, in particular a language dialogue system (language assistant), and translates sign language into text. The language assistant can thereby behave as if the user were speaking to the language assistant in language. The image detection device is mounted at the front, for example, in the dashboard or rearview mirror, and thereby provides precise user monitoring.

[0035] Advantageously, the invention can provide that the image capture device is designed as an infrared camera to optically detect at least one gesture of the vehicle occupant. The camera can be designed, for example, for detection in the near infrared and / or far infrared range to detect input information particularly reliably and with minimal interference.

[0036] Optionally, it can be provided that the output device comprises at least one display, in particular an LED display, for the vehicle interior in order to output a visual output as a visualization of a response to the input information in sign language, wherein the output device is preferably designed as an instrument cluster of the vehicle. This enables easy recognition of the output in sign language.

[0037] The subject of the invention is also a vehicle having a system according to the invention. Thus, within the scope of the invention, it can be advantageous if the vehicle is designed as an autonomous vehicle, ie in particular as a self-propelled motor vehicle. Such a vehicle can be driven, controlled and parked without the influence of a human driver. In this case, the use of a verbal dialogue system is particularly useful, since none of the vehicle occupants are busy controlling the vehicle.

[0038] It is also advantageous if the vehicle is designed as a motor vehicle, in particular a trackless land vehicle, for example as a hybrid vehicle comprising a combustion engine and an electric machine for traction or as an electric vehicle, preferably with a high-voltage on-board electrical system and / or an electric motor. The vehicle can be designed in particular as a fuel cell vehicle and / or a passenger vehicle. Preferably, in an embodiment of the electric vehicle, no combustion engine is provided on the vehicle, so that the vehicle is driven exclusively by electrical energy.

[0039] According to an advantageous extension of the present invention, it can be provided that the output device is arranged in the interior space of the vehicle, in particular in the area of ​​the center console, and the image detection device is arranged in the area of ​​the instrument cluster and / or the center console and / or the interior rearview mirror and / or the rearview mirror and / or the exterior rearview mirror of the vehicle. Here, the image detection device can be arranged on the vehicle in particular so as to be able to detect the driver of the vehicle. Here, for example, it is reasonable to arrange the image detection device at the rearview mirror and / or the exterior rearview mirror on the side of the vehicle where the driver of the vehicle is also sitting. In addition, it is also conceivable that the detection of the image detection device is extended to other occupants of the vehicle. Then, alternative arrangements in or on the vehicle are also suitable accordingly, and / or the use of other image detection devices is also suitable. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Further advantages, features and details of the invention are apparent from the following description, in which embodiments of the invention are described in detail with reference to the accompanying drawings. The features mentioned in the claims and in the description may reflect the essence of the invention individually or in any combination. Among them:

[0041] Figure 1 A schematic diagram of a portion of the system according to the invention is shown from the perspective of the interior of a vehicle;

[0042] Figure 2 A schematic diagram is shown for visualizing the method according to the invention.

[0043] In the following figures, the same reference numerals are used for the same technical features, also for the same technical features of different exemplary embodiments. DETAILED DESCRIPTION

[0044] exist Figure 1 In FIG. 1 , a system 100 for providing a verbal dialogue in sign language of a vehicle 1 is schematically presented. The presentation is here made from the perspective of a vehicle occupant 2 in the vehicle interior 40. An image acquisition device 110 is shown, which can be used to detect input information of the vehicle occupant 2. To this end, the image acquisition device 110 monitors the vehicle interior 40 optically in order to visually record the vehicle occupant 2. At the same time, other vehicle occupants 2 can also be detected by the image acquisition device 110, if necessary.

[0045] A processing device 120 is also provided for evaluating the detected input information, wherein the processing device 120 is designed, for example, as a vehicle electronic device. It is also possible that the evaluation is partially carried out by external components, such as an external data processing device. For this purpose, the processing device communicates with the external data processing device, for example, via a network such as a cellular radio network and / or the Internet. In this way, additional resources can be used for the evaluation.

[0046] An output device 130 may also be provided for visual output in sign language as a function of the evaluation. Usually, in the case of a verbal dialogue system, not only the user inputs in spoken language but also the system 100 responds in spoken language (and / or vice versa). According to the invention, at least one response can be implemented in sign language by the system 100. This thus enables communication with the vehicle 1 even if communication via spoken language is not possible.

[0047] Image capture device 110 can be designed as a camera, in particular an infrared camera, in order to optically detect at least one gesture of vehicle occupant 2 by means of input information. In this case, the gesture is, for example, part of a sign language. In addition, the input information can also include at least one information about a facial expression and / or an unspoken word and / or a body posture of the vehicle occupant, which can also be part of the sign language. This makes it possible to digitally detect the sign language based on this information.

[0048] Furthermore, the gesture detected by the input information can also be a further gesture that is specific to the additional information. This makes it possible to detect the additional information in addition to the information about the sign language. The additional gesture is, for example, pointing to an object 5 (pointing is presented with the aid of a finger of the vehicle occupant 2, thereby providing directional information). The additional information is then information about which object 5 is pointed at. For example, in particular when the input information also includes information about the object 5 or the external area of ​​the vehicle 1, the additional information can also be determined by evaluating the input information; or the additional information can also be determined by evaluating the detection of at least one other image detection device (for example of the external area of ​​the vehicle 1). This makes it possible to evaluate the instructions or questions conveyed by means of sign language during the evaluation and compare the instructions or questions with the additional information. Then, for example, information about the pointed object 5 can be determined as a response.

[0049] Output device 130 may include at least one display 132 , in particular an LED display, for vehicle interior 40 in order to output visual output 131 as a visualization of a response to input information in sign language, wherein output device 130 is preferably designed as instrument cluster 10 of vehicle 1 .

[0050] In addition, the output device 130 can be arranged in the vehicle interior space 40, in particular in the area of ​​the center console 30, and the image detection device 110 can be arranged in the area of ​​the instrument cluster 10 and / or the center console 30 and / or the rearview mirror 21 and / or the exterior rearview mirror 20 of the vehicle 1.

[0051] exist Figure 2 1 shows a method according to the invention for providing a speech dialogue in sign language in a speech dialogue system of a vehicle 1. According to a first method step 101, an optical detection of input information of a vehicle occupant 2 can be performed. According to a second method step 102, an evaluation of the detected input information is performed. According to a third method step 103, a visual output 131 in sign language can then be provided as a function of the evaluation.

[0052] The above description of the embodiments only describes the present invention within the framework of examples. Of course, as long as it is technically reasonable, the various features of these embodiments can be freely combined with each other without departing from the protection scope of the present invention.

[0053] List of Reference Numerals

[0054] 1 Vehicles and motor vehicles

[0055] 2 Vehicle occupants

[0056] 3 Surrounding environment

[0057] 5 Objects

[0058] 10 Instrument cluster

[0059] 20 Exterior mirrors

[0060] 21 Rearview mirror

[0061] 30 Center console

[0062] 40 Vehicle interior space

[0063] 100 Systems

[0064] 101-103 Methods and Steps

[0065] 110 Image detection device

[0066] 120 Processing device

[0067] 130 Output device

[0068] 131 Visual Output

[0069] 132 Display

Claims

1. A method for providing a speech dialogue in sign language in a speech dialogue system of a vehicle (1), The following steps are performed: - performing optical detection of input information from a vehicle occupant (2); - performing an evaluation of the detected input information; - providing a visual output in sign language based on said evaluation (131), in, The execution of the evaluation comprises the following steps: - performing a detection of an interaction of the vehicle occupant (2) with the language dialogue system in sign language on the basis of the input information, wherein the input information is implemented as an image recording of the vehicle occupant (2) in order to detect the interaction as a gesture of the vehicle occupant (2) on the basis of the image recording; - only if the detection of the interaction is successful, an analysis of the input information is performed for the purpose of translating the sign language into content-evaluable translation information for the language dialogue system; - performing, as a function of the input information, detection of a further gesture of the vehicle occupant (2), the further gesture being specific to the additional information; - generating a response in dependence on the translation information and in dependence on the additional information, so as to output the response in sign language via the visual output (131), The response includes the following information: - object information about an object (5) in the surroundings (3) of the vehicle (1), wherein the additional information provides for a selection of the object (5).

2. The method according to claim 1, It is characterized in that The translation information is text information.

3. The method according to claim 1 or 2, It is characterized in that The response also includes at least one of the following information: - Weather information; -Surrounding environment information; - News information; - Media information, In order to generate the response, the information is filtered with regard to the availability of output of the information in sign language.

4. The method according to claim 3, It is characterized in that The selection is a directional description of the object (5).

5. The method according to claim 3, It is characterized in that The surrounding environment information is information about the current location of the vehicle (1).

6. A system (100) for providing a speech dialogue in a sign language for a vehicle (1), the system comprising: - an image detection device (110) for detecting input information of a vehicle occupant (2); - processing means (120) for evaluating the detected input information; - output means (130) for producing a visual output in sign language based on said evaluation, in, The system (100) is designed as a language dialogue system according to the method according to any one of claims 1 to 5 in order to perform the steps of the method.

7. The system (100) according to claim 6, It is characterized in that The image capture device (110) is designed as an infrared camera in order to optically capture at least one gesture of the vehicle occupant (2).

8. The system (100) according to claim 6 or 7, It is characterized in that The output device (130) comprises at least one display (132) for the vehicle interior (40) in order to output the visual output (131) as a visualization of a response to the input information in sign language, wherein the output device (130) is designed as an instrument cluster (10) of the vehicle (1).

9. The system (100) according to claim 8, It is characterized in that The display (132) is an LED display.

10. A vehicle (1) having a system (100) according to any one of claims 6 to 9.

11. The vehicle (1) according to claim 10, It is characterized in that The vehicle (1) is designed as an autonomous vehicle (1).

12. The vehicle (1) according to claim 10 or 11, It is characterized in that The output device (130) is arranged in the vehicle interior space (40), and the image detection device (110) is arranged in the area of ​​the instrument cluster (10) and / or the center console (30) and / or the rearview mirror (21) of the vehicle (1).

13. The vehicle (1) according to claim 12, It is characterized in that The output device (130) is arranged in the area of ​​the central console (30).

Images