Device for communicating with people, especially deaf-mute people, while operating a vehicle

The vehicle system uses cameras and microphones to recognize and translate sign language into optical or acoustic signals, addressing communication and control challenges for deaf and mute individuals, enabling seamless interaction and participation in autonomous vehicles.

DE102022102596B4Active Publication Date: 2024-12-05AUDI AG
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
DE102022102596
Authority / Receiving Office
DE · DE
Patent Type
Patents
Current Assignee / Owner
Filing Date
2022-02-03
Publication Date
2024-12-05
Estimated Expiration
2042-02-03

AI Technical Summary

Technical Problem

Deaf and mute individuals face challenges in communicating and controlling vehicle functions due to the lack of voice control capabilities, and different sign languages can hinder communication among them, limiting their participation in autonomous vehicle technology.

Method used

A vehicle system equipped with cameras and microphones to recognize hand and finger movements, analyze sign language, and convert it into optical or acoustic signals, enabling communication and control through a computing unit that can distinguish between multiple sign languages and translate speech.

Benefits of technology

Facilitates comfortable and effective communication and control of vehicles for deaf and mute individuals, allowing them to interact with both hearing individuals and other mute individuals using different sign languages, enhancing their participation in autonomous mobility.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 00000000_0000_ABST
    Figure 00000000_0000_ABST
Patent Text Reader

Abstract

Device (50) for communicating with persons (7, 8), in particular deaf-mute persons (7), when operating a vehicle (1) with - the vehicle (1) which has at least one camera (3) and at least one microphone (4) which are permanently installed in the vehicle (1), wherein the camera (3) is designed to detect individual fingers of a hand and their movements and the microphone (4) is designed to detect acoustic signals caused by the movements of the hand and / or fingers, - an output unit (6) in the vehicle (1) for outputting optical and / or acoustic signals, - a computing unit (2) which is designed, - to analyze the movements of the hand, fingers and / or the recorded acoustic signals caused by the movements of the hand and / or fingers to recognize a sign language, - and, depending on this analysis, to provide an optical and / or acoustic signal by means of the output unit (6) which corresponds to the recognized sign language, characterized by - an acoustic unit wearable on the hand or finger, which is designed to emit predetermined sound waves which can be detected by the at least one microphone (4), wherein - the computing unit (2) is set up to perform the analysis for recognizing sign language additionally depending on the sound waves recorded.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] The invention relates to a vehicle for communicating with people, especially deaf and mute people. The invention further provides a corresponding method for communicating with deaf and mute people.

[0002] As a result of technological advances and globalization, more and more people are gaining access to mobility and technology. Deaf-mute people also belong to this group. Deaf-mute people have fewer options for communication. Such people often use sign language to communicate with the outside world. With increasing autonomous mobility, deaf-mute people cannot benefit from certain technological innovations. For example, it is not possible for deaf-mute people to control vehicle functions using voice control. Until now, deaf-mute people have had to press buttons or gearshifts inside a vehicle to operate vehicle functions.

[0003] However, deaf-mute people also want to participate in technological progress. Autonomous or semi-autonomous vehicle operation is also a future field that should not be closed to deaf-mute people.

[0004] A particular challenge can arise when several deaf-mute people who use different sign languages ​​meet. Since there are approximately 300 different sign languages, two deaf-mute people may meet and yet still be unable to communicate using their respective sign languages.

[0005] The publication "Implementation of Virtual Assistant with Sign Language using Deep Learning and TensorFlow" by Dipanshu Someshwar, Dharmik Bhanushali, and Vismay Chaudhari, published in the Proceedings of the Second International Conference on Inventive Research in Computing Applications (ICIRCA-2020) with ISBN 978-1-7281-5374-2, describes a system that enables deaf-mute people to use virtual assistants using sign language. Hand gestures are captured using a webcam. A neural network interprets the captured images and creates rational languages. These languages ​​are mapped to predefined datasets using deep learning.

[0006] The publication "Sign Language Recognition, Generation, and Translation: An Interdisciplinary Perspective" by Danielle Bragg et al. from the ASSETS' 19 conference, October 28-30, 2019, Pittsburgh, PA, USA, focuses on sign language processing. It presents concepts that translate signed content into written queries.

[0007] The generic document DE 11 2016 006 769 T5 discloses sensors in a vehicle that detect the execution of a sign language symbol. The sign language symbol invokes a vehicle interface function. The sensors include three-dimensional optical, ultrasonic, or other sensors. The locations of targets (fingertips, knuckles, palm, etc.) are detected in the individual sensor outputs. The locations of targets determined from the outputs of multiple sensors can be associated with each other and filtered.

[0008] Furthermore, document CN 2 03 819 120 U discloses a device for assisting a deaf person when riding a bus or taking a taxi. The device comprises a control unit, a camera arranged on a passenger compartment console of a vehicle and used to capture the deaf person's sign language image information, a loudspeaker for transmitting an audio signal, and a speech chip for driving the loudspeaker to generate a sound. Speech data corresponding to various sign language image information is preset in the control unit. The camera is connected to the input of the control unit. The speech chip is connected to the output of the control unit. The loudspeaker is connected to the output of the speech chip. By inserting the device, the deaf person can communicate with the regular driver to take the bus and can easily select the driving mode.

[0009] Furthermore, publication DE 10 2019 204 054 A1 describes a method for providing a speech dialogue in sign language in a speech dialogue system for a vehicle. Input information from a vehicle occupant is optically captured, the captured input information is evaluated, and a visual output in sign language is provided depending on the evaluation.

[0010] Furthermore, the document DE 10 2006 017 509 A1 discloses a device for translating sign language with a camera and loudspeaker, with a computer housed in a housing to be worn around the neck, which determines predetermined word sequences from the signals of the one or more cameras arranged on the front of the housing by evaluating the recorded movements of the arm(s) of the sign language user and transmits them to the loudspeaker.

[0011] One object of the invention can be seen in enabling deaf-mute persons to use vehicles more comfortably. This object is achieved by means of the independent patent claims. Useful further developments and alternative embodiments are apparent from the dependent claims, the description, and the figures.

[0012] A first aspect of this invention provides a device for communicating with people, particularly deaf-mute people. The vehicle may accommodate people who can hear and speak, as well as people who are unable to do so. Communication or interaction between normal people and deaf-mute people, or between the vehicle and the deaf-mute people, is often limited. Currently, deaf-mute people can usually only interact with the vehicle by operating mechanical buttons, control or steering levers, or by operating a touchscreen display. This invention is intended to improve communication between deaf-mute people and other people or the vehicle.

[0013] The device preferably comprises a vehicle. The vehicle, in particular, has at least one camera and at least one microphone. The at least one camera and the at least one microphone are preferably each permanently installed in the vehicle. The camera is preferably designed to recognize individual fingers of a hand and their movements. This means, in particular, that the camera's resolution is high enough to recognize a hand movement as well as the movement of individual fingers of the hand. In many cases, a commercially available webcam or smartphone camera is sufficient for this purpose.

[0014] The microphone is particularly designed to detect acoustic signals caused by the movements of the hand and / or fingers. Additionally or alternatively, the microphone can be configured to detect speech sound, in particular spoken words. Speech sound can be understood as phonetic sounds, word content and / or linguistic sounds. The microphone can in particular be a Doppler microphone, which is capable of registering a frequency change of the acoustic signals. With the aid of the detected frequency change or the detected acoustic signals, the proximity or distance of the hand and / or fingers to the microphone can be determined. For example, acoustic signals can be generated by snapping the fingers or clapping or tapping the hand. These acoustic signals are preferably detected using the at least one microphone.However, it is also possible that the microphone can detect sound pressure caused by hand and / or finger movement.

[0015] The vehicle preferably has an output unit for outputting visual and / or acoustic signals. The output unit can be a display, a touchscreen (touch display), or a monitor for outputting visual signals. Additionally or alternatively, the output unit can be a radio or loudspeaker for providing acoustic signals. Acoustic signals can be useful for communication between deaf and hearing people.

[0016] In particular, the vehicle has a computing unit configured to analyze the movements of the hand, fingers, and / or the detected acoustic signals caused by the movements of the hand and / or fingers in order to recognize a sign language. The computing unit can preferably distinguish between several different sign languages. To do so, the computing unit can, for example, compare the detected movements and / or the detected acoustic signals with predefined movement patterns and / or audio patterns of one or more sign languages.

[0017] Additionally or alternatively, the computing unit can be configured to analyze the captured speech sound to recognize speech content. This is preferably the case with a voice command.

[0018] Depending on this analysis, the computing unit can generate a control signal for the output unit. The control signal is preferably configured to provide a visual display via the output unit that corresponds to the recognized sign language. This means, in particular, that the computing unit assigns the recognized movements and / or detected acoustic signals to a sign language and then converts them into a corresponding visual signal. Thus, the computing unit can transcribe sign language from deaf-mute people into visual signals. The visual display of the display preferably corresponds to a semantic content of the recognized sign language. Additionally or alternatively, the display can be designed as an audio output unit. This means that the computing unit can convert the recognized sign language into an audio signal. For this purpose, the vehicle can have the output unit, for example in the form of a loudspeaker.For example, sign language can be recorded using at least one camera and at least one microphone. The processing unit can transcribe the sign language into a visual and / or acoustic signal. This allows deaf-mute people to communicate with other people who can hear and speak.

[0019] Alternatively, the control signal can be designed solely to control a vehicle function. For example, the control signal can provide control of an air conditioning system to adjust the temperature in the vehicle. In this case, the control signal is additionally displayed visually and / or acoustically with the help of the output unit. For example, a telephone function can be requested using sign language, which can be visually confirmed using the display. If speaking people want to communicate with deaf-mute people, the deaf-mute person cannot perceive the spoken speech sound. The processing unit can serve as an "interpreter" by translating the speech sound into sign language and vice versa.

[0020] Speech sound can thus be a spoken word sequence that has a predetermined meaning. This predetermined meaning corresponds to the speech content. The processing unit can analyze the detected speech sound to recognize the speech content. Depending on this analysis, the processing unit can generate an optical and / or acoustic signal using the output unit. This optical and / or acoustic signal preferably corresponds to the recognized sign language or the detected speech sound. In particular, the optical and / or acoustic signal can reproduce the speech content in the form of sign language.

[0021] The processing unit can have access to relevant information and / or a corresponding database for analysis. The processing unit can assign the recorded hand and / or finger movements to a predefined movement pattern from a corresponding sign language. This allows the processing unit to assign the hand and / or finger movements as well as the recorded acoustic signals to a sign language and subsequently convert these recorded movements and / or acoustic signals into optical and / or acoustic signals.

[0022] The following different application scenarios are conceivable as examples within the scope of this invention: For example, a deaf-mute person can operate an autonomous or semi-autonomous vehicle. Using sign language, the deaf-mute person can express control signals for controlling predefined vehicle functions. The processing unit can capture the sign language and convert it into a corresponding control signal. The output unit can provide optical confirmation of the activation or deactivation of the requested vehicle function. Since in this case the signal from the output unit is directed at a deaf-mute person, the output signal is preferably optical. The optical signal can be in the form of text or writing, or in the form of sign language.

[0023] Another application is communication between a deaf-mute person and a non-deaf-mute person (able to speak and hear) in a vehicle. The non-deaf-mute person is referred to below as the occupant. If the deaf-mute person wishes to communicate with other occupants, the sign language of the deaf-mute person can be recorded, analyzed, and translated into corresponding visual and / or acoustic signals for the other occupants using at least one camera and at least one microphone. With the help of the output unit, the sign language of the deaf-mute person can be output as appropriate visual and / or acoustic signals for the other occupants. For example, the deaf-mute person can use sign language to communicate that they are hungry, for example.With the help of the output unit, this declaration of intent expressed through sign language can be issued in text form and / or in audible speech for the other occupants.

[0024] Accordingly, in another application, the reverse communication path can also be used. In this case, the inmate who can speak and hear can communicate with the deaf-mute person. In this application, the preferred approach is to convert acoustic signals and speech sounds into sign language. With the help of at least one microphone, the spoken speech sound of the inmate or the word content based on the speech sound can be recorded. The computing unit is particularly designed to analyze this speech sound and assign it to a speech content. The speech content recognized in this way can be converted by the computing unit into an optical signal which corresponds to the speech content of the speech sound. The deaf-mute person can perceive this optical signal. The optical signal can be in the form of text, sign language and / or color, for example.

[0025] Another application scenario would be the presence of two deaf-mute people who, due to different sign languages, are unable to communicate with each other. In this case, the vehicle's processing unit can serve as an interpreter. The sign language of the first deaf-mute person, as well as the acoustic signals evoked by the sign language movements, can be recorded and analyzed. This analysis preferably involves capturing the content of the sign language. The processing unit can translate this content, or "sign content," into another sign language suitable for the second deaf-mute person.

[0026] The processing unit can, with the aid of the output unit, output an optical signal corresponding to the second sign language and reflecting the content of the first sign language. Accordingly, the processing unit can additionally or alternatively translate the content of the second sign language into the first sign language and output it accordingly using an associated optical signal.

[0027] In principle, it doesn't matter whether the people involved are inside or outside the vehicle. This makes it possible for a deaf-mute person outside the vehicle to communicate with another person inside. Depending on the application, the processing unit can output optical, acoustic, or both optical and acoustic signals via the output unit.

[0028] Thus, the vehicle with at least one camera, at least one microphone, output unit and computing unit can be used in a variety of ways to meet the needs of deaf and mute people.

[0029] An additional or alternative advantageous embodiment provides a device with an ultrasonic sensor. The ultrasonic sensor can be part of a gesture control unit. The gesture control unit can be a touch display in the area of ​​the dashboard. The computing unit is, in particular, capable of accessing sensor data from the ultrasonic sensor. Furthermore, the computing unit is preferably designed to use the ultrasonic sensor to determine a dynamic change in the distance of the hand and / or fingers from the ultrasonic sensor. Furthermore, the computing unit is, in particular, designed to perform the analysis additionally based on the dynamic change in the distance for recognizing the sign language and / or a content of the sign language.

[0030] This allows the spatial distance between the hand or fingers and the ultrasonic sensor to be recorded, and this dynamically variable distance can be used to determine a trajectory of the hand. Using this trajectory, the computing unit can recognize sign language and derive the content of the sign language from it. The ultrasonic sensor can be arranged in the area of ​​a dashboard and / or in the area of ​​a roof liner. In addition, a reflection unit can be arranged on a finger or in the area of ​​a forearm or wrist, which reflects ultrasonic signals from the ultrasonic sensor better than a hand or arm without the reflection unit. For example, the reflection unit could be a small metal ring or a metal bracelet. The additional sensor data that can be extracted in this way can help the computing unit to better analyze the movements of the hand and / or fingers for the purpose of recognizing sign language.In particular, this additional information from the ultrasonic sensor can enable more detailed differentiation between several different sign languages. In particular, several ultrasonic sensors can be distributed throughout the passenger seating area of ​​the vehicle.

[0031] It is provided that the vehicle has an acoustic unit that can be worn on the hand, for example in the form of a ring. Since deaf-mute people usually have two hands, it is preferably provided that the vehicle or the computing unit can interact with two acoustic units, each of which can be arranged on a hand, in particular on a finger. The acoustic unit or the ring is preferably designed to emit predetermined sound waves. These sound waves can be detected by the at least one microphone, and the computing unit is configured to additionally carry out the analysis for recognizing sign language depending on the detected sound waves. The emitted sound waves preferably always have the same volume. This makes it possible to calculate the distance of the hand to the at least one microphone based on a volume measurement that can be determined by the microphones.

[0032] This can help determine the trajectory of the fingers and / or hand. The trajectory can be assigned to a corresponding sign language by the processing unit. At the same time, the processing unit can output the corresponding sign language content. This sign language content can be presented visually and / or acoustically to other people or passengers using the output unit. If a deaf-mute person wears at least one ring on each hand that emits the specified sound waves, this can help the processing unit recognize the sign language more reliably or quickly.

[0033] Additionally or alternatively, it can be provided that the specified sound waves have a frequency of less than 15 hertz and / or a frequency of more than 21,000 hertz, i.e. 21 kilohertz. Frequencies below 15 hertz can be considered infrasound, while frequencies above 21 kilohertz can be referred to as ultrasound. These frequency ranges represent a range that is inaudible to humans. If, for example, a deaf-mute person wears one or more rings that emit sound waves in this frequency range, these emitted sound waves will not be perceived by other hearing-capable people. This can further refine the recognition of sign language because the computing unit can access additional sensor data with the help of the emitted sound waves, while at the same time acoustic impairment for other people can be avoided.In this case, only at least one microphone must be designed to capture sound waves that are inaudible to humans.

[0034] It is possible for these rings to also emit sound waves within the audible range to test their functionality as acoustic sound waves. It is also possible to provide a corresponding ring for each individual finger as an acoustic source. In this case, the device can be expanded to include up to 10 acoustic units. The acoustic unit can be a ring that can emit tones or sound waves. Thus, the device can include multiple rings per hand to enable more reliable recognition of sign language by the processing unit.

[0035] An additional or alternative embodiment provides a device with a wrist-wearable smartwatch, which has a position sensor, an acceleration sensor, and a jolt sensor. The smartwatch is preferably arranged on a wrist or forearm of the deaf-mute person. The smartwatch can transmit sensor data to the computing unit using said sensors. This preferably occurs wirelessly. For this purpose, both the computing unit and the smartwatch can have a corresponding interface. The computing unit is preferably configured to determine a trajectory of the hand based on sensor data from these sensors and to consider the determined trajectory for the analysis to recognize the sign language. The computing unit can thus access the sensors provided by the wearable smartwatch in order to better and more reliably recognize the sign language or the content of the sign language.

[0036] The acceleration sensor can be used to distinguish between hand movements of different speeds in sign language. The same applies to the information provided by a jerk sensor. Using the sensor data from the position sensor, the acceleration sensor, and the jerk sensor, the processing unit can determine the trajectory of the hand or the respective fingers in detail. The trajectory of the hand or the respective fingers thus determined can be compared or compared with predefined movement patterns. In this way, the processing unit can assign the recorded movements to sign language with the help of the processing unit. This allows the processing unit to extract content from the sign language. This is achieved with the help of the determined trajectory, which can be compared with predefined movement patterns.For this purpose, the processing unit can access stored, predefined movement patterns and / or an external database. In the case of an external database, the device can have a corresponding interface to establish a data connection to the external storage unit containing the movement patterns stored there.

[0037] An additional or alternative embodiment provides a device with multiple rows of seats in the vehicle. Each row of seats has at least two cameras and at least three microphones, which are distributed within the row of seats. The cameras and microphones can be designed as one or more microphone-cameras. Each individual seat in the respective row of seats is preferably arranged within a detection range of the cameras and microphones. By appropriately synchronizing the cameras with the microphones, the location of a sound source can be visually visualized. This means that with the appropriate interaction of the at least three microphones and at least two cameras, it is possible to localize acoustic signals. The computing unit can thereby determine which seat is being spoken from.This allows the processing unit to preferentially increase the sensitivity of those sensors intended for the respective seat. This can further improve communication between passengers and deaf-mute people.

[0038] An additional or alternative embodiment provides a device with a vibration module, wherein the vibration module is configured to provide a haptic signal to the person or occupant depending on the analysis performed by the computing unit. For example, the output unit can contain the vibration module. The vibration module can be arranged in the seat, for example. With the aid of a haptic signal, the computing unit can provide rapid confirmation. This applies in particular to binary statements. A binary statement is, for example, an answer to a question that can be answered with "yes" or "no," or positively or negatively.

[0039] For example, the deaf-mute person could use sign language to ask whether a previously conducted coronavirus test was positive or negative. In this case, the processing unit can establish a data connection to the relevant testing center and retrieve the result there. The processing unit can provide the requested test result in the form of an optical signal via the output unit. Alternatively, the processing unit can use the vibration module to generate a vibration that the deaf-mute person can perceive. For example, a pleasant vibration can indicate a negative test result.

[0040] An additional or alternative embodiment provides a device in which the computing unit includes a neural network. The neural network is particularly designed to learn to assign the recorded movements of the hand and / or fingers as well as the acoustic signals caused by the movements of the hand and / or fingers to a semantic content and to display them visually, acoustically and / or haptically with the aid of the output unit. The haptic signal can be a vibration. In particular, the neural network can learn to classify new, unknown movements of the hand and / or fingers and to learn their meaning. To this end, the neural network can compare the recorded movements with already known movement patterns. The comparison can, for example, include a similarity analysis. Deaf-mute people can thus specify new, individual movements that the computing unit can learn.This allows the processing unit to be conditioned to specific movements of the deaf-mute person. This can be helpful, for example, in an emergency situation.

[0041] For example, the deaf-mute person can predetermine a predetermined movement, which the processing unit interprets as an SOS or emergency signal. This predetermined movement of the hand and / or fingers is preferably unnoticeable to other people, but can generate an emergency signal. In this case, the processing unit can send out a corresponding emergency signal to the outside world. This can be particularly advantageous in the event of a kidnapping, as the deaf-mute person can summon help unnoticed. With the help of a suitable position sensor, the processing unit can also regularly transmit the vehicle's position data along with the emergency signal. In this way, a deaf-mute person can trigger a type of "silent alarm" in an emergency situation.

[0042] An additional or alternative embodiment provides a device having an internal vehicle storage unit and / or an external storage unit. Predefined movement patterns of the hand and / or fingers are preferably stored on the internal vehicle storage unit and / or external storage unit. The computing unit is preferably designed to compare and / or evaluate the recorded movement data with the movement patterns stored on the internal vehicle storage unit and / or external storage unit for the analysis. For this purpose, the computing unit can have a corresponding neural network. With the help of the neural network, new movement patterns can be recognized and added to a corresponding database. New movement patterns can be analyzed or examined better and in more detail externally, for example with the help of a data center.The device and / or the neural network can even learn to recognize a new sign language.

[0043] A second aspect of this invention relates to a method for communicating between a vehicle and a person, particularly a deaf-mute person. The method particularly comprises the following method steps: In a first step a), at least one camera and at least one microphone are arranged or provided. The cameras and microphones are arranged, in particular, in an interior of the vehicle. The cameras are preferably designed to detect individual fingers of a hand and their movements, while the microphone is designed to detect acoustic signals based on the movements of the hand, the fingers, and / or speech sound.

[0044] In a further step b), hand and finger movements can be recorded. Additionally or alternatively, acoustic signals and speech sounds can be recorded. The recorded hand, finger movements, acoustic signals, and / or speech sounds are preferably analyzed in step c) to recognize sign language. This analysis is preferably performed by a computing unit. The computing unit can comprise a neural network.

[0045] Depending on the analysis, an optical, haptic, and / or acoustic signal can be generated in step d). The signal can be provided by an output unit. The signal corresponds to the recognized sign language or the recorded speech sound.

[0046] Features, advantages, and exemplary embodiments mentioned in connection with the first aspect apply mutatis mutandis and analogously to the second aspect of the invention, and vice versa. Device features can thus be interpreted as method features, and vice versa. The vehicle can comprise a computer program containing instructions that cause each embodiment of the method to be executed. The computer program can be stored on a computer-readable medium. It can be stored in the computing unit or the neural network.

[0047] The invention will now be explained in more detail with reference to the accompanying drawings. It should be noted that the drawings merely represent exemplary embodiments of the invention. All examples mentioned in the figures can be combined with one another, provided they are not mutually exclusive.

[0048] The invention also includes the computing unit for the vehicle. The computing unit can have a data processing device or a processor device configured to carry out an embodiment of the method according to the invention. For this purpose, the processor device can have at least one microprocessor and / or at least one microcontroller and / or at least one FPGA (Field Programmable Gate Array) and / or at least one DSP (Digital Signal Processor). Furthermore, the processor device can have program code configured to carry out the embodiment of the method according to the invention when executed by the processor device. The program code can be stored in a data memory of the processor device.

[0049] The invention also includes further developments of the method according to the invention that have features already described in connection with the further developments of the motor vehicle according to the invention. For this reason, the corresponding further developments of the method according to the invention are not described again here.

[0050] The vehicle according to the invention can be a motor vehicle, a motor vehicle, in particular a passenger car or a lorry, an aircraft, a passenger bus or a motorcycle.

[0051] The invention also encompasses combinations of the features of the described embodiments. The invention therefore also encompasses implementations that each comprise a combination of the features of several of the described embodiments, unless the embodiments are described as mutually exclusive.

[0052] Exemplary embodiments of the invention are described below. Shown are: Fig. 1 a first embodiment with two persons in the vehicle; Fig. 2 a second embodiment with a deaf-mute person outside the vehicle.

[0053] The exemplary embodiments explained below are preferred embodiments of the invention. In the exemplary embodiments, the described components of the embodiments each represent individual features of the invention that can be considered independently of one another, each of which also develops the invention independently of one another. Therefore, the disclosure is intended to encompass combinations of the features of the embodiments other than those shown. Furthermore, the described embodiments can also be supplemented by further features of the invention already described.

[0054] In the figures, the same reference symbols designate elements with the same function.

[0055] In Fig. 1 shows an example of a vehicle 1 with several people 7, 8. A deaf-mute person 7 can be seen behind a steering wheel on the left. To the right of this person 8 can be seen. It is assumed that person 8 can hear and speak, while the deaf-mute persons 7 cannot. Fig. The device 50 shown in Figure 1 is designed to allow the deaf-mute persons 7 to communicate with both the vehicle 1 and the person 8. For this purpose, the vehicle 1 has a processing unit 2, several microphones 4, and several cameras 3. A microphone 4 and a camera 3 can be seen in the front row of seats. The same applies to the rear row of seats, in which only a single person, namely the deaf-mute person 7, is located. The vehicle 1 has additional gesture control units. The gesture control units can be designed as ultrasonic sensors 5. Hand movements and / or finger movements can be detected with the help of the gesture control units.

[0056] The cameras 3 are particularly designed to capture sign language from deaf-mute persons 7. This means that the cameras 3 can recognize hand movements and even individual finger movements. For this purpose, the cameras 3 can have a resolution of at least 1 megapixel. With the help of the microphones 4, acoustic signals caused by the hand movements of the sign language can be captured. Likewise, voice commands or speech sounds can be recorded with the help of the microphones 4. The vehicle 1 further has an output unit 6. The output unit 6 can be a screen, a monitor, a display and / or an acoustic output unit. The output unit 6 can include a loudspeaker. Thus, both acoustic and optical signals can be output with the help of the output unit 6.

[0057] Optical signals are preferably output for the deaf-mute persons 7. The multiple microphones 4, multiple cameras 3, and gesture control units (ultrasonic sensors 5) can relate to information or sensor data that detect hand movements and / or finger movements. Using this information, the computing unit 2 can analyze the movements of the hand, the fingers, and / or the detected acoustic signals and thus recognize sign language. In particular, the computing unit 2 can have a neural network 2'. Using the neural network 2', the detected movements of the hand and / or fingers can be translated into optical and / or acoustic signals, which preferably correspond to a sign language content. Using the output unit 6, the content of the sign language can, for example, be displayed or read aloud to the other person 8.

[0058] Likewise, the computing unit 2 can capture the word content of the other person 8 and translate it into sign language. For this purpose, the computing unit 2 can capture and analyze the captured speech sound of the other person 8. As part of this analysis, a speech content can be derived from the speech sound. For this purpose, the computing unit 2 can access the neural network 2'. With the help of the neural network 2', individual words can be determined from the speech sound and assigned to a speech content. This speech content can be transcribed into a corresponding sign language and displayed in optical form for the deaf-mute persons 7 using the output unit 6. This means that the deaf-mute persons 7 in the example of Fig. 1 are presented with the corresponding sign language on their respective screens.

[0059] The device 50 can further comprise a vibration module 9. With the aid of the vibration module 9, the computing unit 2 can provide a haptic signal to the deaf-mute person 7. For example, the computing unit 2 can confirm the deactivation or activation of a vehicle function to the deaf-mute person 7 using a slight vibration. Haptic signals can be suitable for providing binary responses. Binary responses are, in particular, responses that can be answered with "yes" or "no." With the aid of the device 50, the deaf-mute person 7 can access additional vehicle functions.

[0060] For example, a connection can be established to an external storage unit 10 to stream videos or films. The computing unit 2 can also access information from the external storage unit 10 to analyze the recorded movements of the hand and / or fingers. The external storage unit 10 can contain information regarding predefined movement patterns. The movement patterns stored on the external storage unit 10 can be compared and / or aligned with the movement patterns recorded by the cameras 3 and microphones 4. In conjunction with the external storage unit 10, additional information can be provided for analyzing and evaluating the sign language. This can enable more efficient interaction or communication with the deaf-mute persons 7.With the aid of the device 50, the deaf-mute persons 7 can, for example, be enabled to make telephone calls using sign language, to give vehicle commands, to communicate with the other person 8, and / or even to consume audio content.

[0061] For the consumption of audio content, the processing unit 2 can receive signals from a radio station and translate these radio signals into sign language, displaying them visually via the output unit 6. The vehicle 1 with such a device 50 can represent significant added value for deaf-mute persons 7. The multiple microphones 4 can record sound waves generated by hand movements. In particular, frequency shifts due to the physical Doppler effect can be registered with the aid of the multiple microphones 4. As a result, trajectories relating to the hand movements and / or finger movements can be determined with the aid of the multiple microphones 4 and the multiple cameras 3. These trajectories can be analyzed and evaluated to recognize sign language.

[0062] Additionally, the multiple cameras 3 can be used to record gestures or facial expressions of the deaf-mute person 7. This allows for further refinement of the analysis or recognition of sign language. Different sign languages ​​can also be translated into one another using the processing unit 2. For example, if the deaf-mute person 7 in front uses a first sign language, while the deaf-mute person 7 sitting behind them uses a different sign language, the processing unit 2 can perform a corresponding translation. This allows deaf-mute people who use different sign languages ​​to still communicate or interact with each other.

[0063] The multiple cameras 3 and multiple microphones 4 are preferably arranged within the vehicle 1 in such a way as to capture three-dimensional information relating to the sign language. In particular, a three-dimensional trajectory of the hand and / or fingers can be determined or captured by the appropriate arrangement of the multiple cameras 3 and multiple microphones 4. Corresponding ultrasonic sensors 5 of the gesture control units can also be used for this purpose. These three-dimensional trajectories can be used by the computing unit 2 or the neural network 2' to recognize the sign language and the associated content of the sign language.

[0064] Using the output unit 6, sign language content can be visually displayed. Audio content can also be visually displayed on the output unit 6. The device 50 can even be used by the deaf-mute person 7 to make telephone calls. The deaf-mute person 7 preferably communicates using their corresponding sign language, which is recorded by the multiple microphones 4, the multiple cameras 3, and / or multiple gesture control units 5. The data acquired by these sensors is transmitted to the computing unit 2. With the aid of a database stored on the computing unit 2 and / or with the aid of an internet connection to the external storage unit 10, the analyzed sign language can be converted or provided as written text, an acoustic signal, and / or a haptic signal. The other person 8, on the other hand, can speak normally.Their speech sound can be recorded by the microphones 4 and translated into corresponding sign language by the processing unit 2. The sign language can be visually output to the deaf-mute persons 7 via the corresponding output unit 6.

[0065] In Fig. 2 shows a further embodiment of the device 50. In this case, another deaf-mute person 7 is located outside the vehicle 1. The vehicle 1 can have an outward-facing microphone. The outside deaf-mute person 7 can be detected using the front camera 3. In Fig. 2, the corresponding detection areas of the respective cameras 3 are indicated by dashed lines. Accordingly, the camera 3 next to the computing unit 2 can optically detect the deaf-mute person 7 standing outside. Double arrows representing a sign language are indicated in the area of ​​the arms of the deaf-mute person 7 standing outside. With the help of the corresponding camera 3 and the outward-facing microphone 4, the sign language of the deaf-mute person 7 standing outside can be detected and translated accordingly for the other person 8 or the deaf-mute person 7 sitting inside the vehicle 1. For this purpose, the computing unit 2 can translate the sign language of the deaf-mute person 7 standing outside into a corresponding sign language for the deaf-mute person 7 inside the vehicle, and the sign language thus translated can be optically displayed using the output unit 6.At the same time, the recorded sign language of the external deaf-mute person 7 can be acoustically output to the other person 8 with the help of the corresponding output unit 6.

[0066] In the example of Fig. 2, the other person 8 can be alerted to the deaf-mute person 7 outside by means of the vibration module 9. In Fig.2 shows that the vehicle 1 also has several microphones 4 and several cameras 3 on the side. With the help of a corresponding arrangement of the several microphones 4 and the several cameras 3, an environmental sensor system can be created that completely covers a predetermined area around the vehicle 1. This allows sign language from outside deaf-mute persons 7 to be converted into written or spoken words, which can then be presented in the interior of the vehicle 1 via the output units 6. This creates a good communication channel between police officers or authorities and vehicle occupants.

[0067] The new device 50 can capture sign language and / or speech signals using the multiple cameras 3 and multiple microphones 4. These signals can be translated into sign language or visual and acoustic signals. The computing unit 2 preferably ensures that the respective content of the speech sound or sign language is retained. Thus, sign language can also be recognized within the vehicle 1 and translated into natural speech. This natural speech can be communicated acoustically to the additional person 8 via the output unit 6.

[0068] As a result, the device 50 can also enable deaf-mute persons 7 to participate in traffic more comfortably and pleasantly. In particular, the acceptance of autonomously driving vehicles 1 can be increased among deaf-mute persons 7. The vehicle 1 can be designed as a motor vehicle, passenger car, aircraft, motorcycle, truck, and / or amphibious vehicle. The use or operation of these vehicles 1 can be made easier for the deaf-mute persons 7 with the aid of the device 50.

Claims

[1] Device (50) for communicating with persons (7, 8), in particular deaf-mute persons (7), when operating a vehicle (1) with - the vehicle (1), which has at least one camera (3) and at least one microphone (4) which are permanently installed in the vehicle (1), wherein the camera (3) is designed to detect individual fingers of a hand and their movements and the microphone (4) is designed to detect acoustic signals caused by the movements of the hand and / or the fingers, - an output unit (6) in the vehicle (1) for outputting optical and / or acoustic signals, - a computing unit (2) which is designed - to analyse the movements of the hand, fingers and / or the recorded acoustic signals caused by the movements of the hand and / or fingers for the recognition of a sign language, - and depending on this analysis, to provide an optical and / or acoustic signal by means of the output unit (6) which corresponds to the recognized sign language, characterized by - an acoustic unit which can be worn on the hand or finger and which is designed to emit predetermined sound waves which can be detected by the at least one microphone (4), wherein - the computing unit (2) is configured to additionally carry out the analysis for recognizing the sign language as a function of the recorded sound waves. [2] Device (50) according to claim 1 with an ultrasonic sensor (5), wherein the computing unit (2) is designed to use the ultrasonic sensor (5) to determine a dynamic change in a distance of the hand and / or the fingers to the ultrasonic sensor (5) and additionally to carry out the analysis for recognizing the sign language based on the dynamic change in the distance. [3] Device (50) according to one of the preceding claims, wherein the acoustic unit is designed as a ring. [4] Device (50) according to one of the preceding claims, wherein the microphone (4) is designed to detect speech sound, and the computing unit (2) is designed to analyze the detected speech sound to recognize a speech content. [5] Device (50) according to one of the preceding claims, comprising a smartwatch which can be worn on a wrist or on a forearm and which has a position sensor, an acceleration sensor and a jerk sensor, and the computing unit (2) is configured to determine a trajectory of the hand based on sensor data from these sensors and to take the determined trajectory into account for the analysis for recognizing the sign language. [6] Device (50) according to one of the preceding claims with a plurality of rows of seats in the vehicle (1), wherein in each row of seats at least two cameras (3) and at least three microphones (4) are arranged distributed, which are designed in particular as one or more microphone cameras. [7] Device (50) according to one of the preceding claims, comprising a vibration module (9) which is designed to provide a haptic signal depending on the analysis carried out by the computing unit (2). [8] Device (50) according to one of the preceding claims, wherein the computing unit (2) includes a neural network (2') which is designed to learn to assign the detected movements of the hand and the fingers as well as the acoustic signals caused by the movements of the hand and / or the fingers to a semantic content and to display them optically, haptically and / or acoustically with the aid of the output unit (6). [9] Device (50) according to one of the preceding claims, comprising an internal vehicle storage unit and / or an external storage unit (10) on which predetermined movement patterns of the hand and / or fingers are stored, and the computing unit (2) is designed to compare and / or evaluate the detected movements with the movement patterns stored on the internal vehicle and / or external storage unit (10) for the analysis. [10] Method for communicating a vehicle (1) with a person (7, 8), in particular a deaf-mute person (7), by carrying out the following method steps: a) arranging or providing at least one camera (3) and at least one microphone (4) in an interior of the vehicle (1), wherein the camera (3) is designed to detect individual fingers of a hand and their movements and the microphone (4) is designed to detect acoustic signals based on the movements of the hand and / or the fingers, b) Recording the movements of the hand and fingers and recording the acoustic signals, c) Analyzing the recorded movements of the hand, fingers and acoustic signals to recognize a sign language, d) generating an optical, haptic and / or acoustic signal depending on the analysis, the signal corresponding to the recognized sign language, characterized by e) emitting predetermined sound waves, which can be detected by the at least one microphone (4), by an acoustic unit that can be worn on the hand or finger, and f) Carrying out the analysis to recognize the sign language additionally depending on the recorded sound waves.

Citation Information

Patent Citations

  • Device for assisting deaf in taking bus

    CN203819120U

  • Sign language translating device, has computer accomodated in case that is supported around neck for determining predetermined words orders from signals of multiple cameras under evaluation of determined paths of movement of arms

    DE102006017509A1

  • Method for providing a speech dialogue in sign language in a speech dialogue system for a vehicle

    DE102019204054A1

  • Procedures for sign language input into a vehicle user interface and vehicle

    DE112016006769T5

  • CN000203819120U