Method for supporting hearing understanding of user of hearing device

Abstract

The invention relates to a method for supporting the hearing understanding of a user of a hearing device and to an associated hearing system having such a hearing device. Ambient sounds containing speech are received from a user's surroundings by means of a hearing device. Speech contained in the received ambient sound is automatically converted into text data. The text data is output as a graphical representation of the text on a screen of the hearing device or of a peripheral device connected to the hearing device by means of a data transmission technique and / or as synthetic speech in the form of a sound signal to the user. For the speech contained in the ambient sound, the direction of origin and / or at least one speaker characteristic are / is determined automatically in a time-resolved manner. The graphical representation of the text data or the synthesis of the speech is varied in a time-resolved manner depending on the determined source direction and / or the at least one speaker characteristic. As additional or alternative to immediate output to the user, a graphical representation of the text data or synthesized speech is recorded for later output.

Description

Technical Field

[0001] The present invention relates to a method for supporting the hearing comprehension of a user of a hearing device. Furthermore, the present invention relates to a hearing system having such a hearing device. Background Art

[0002] Electronic devices that support the hearing of a person wearing a hearing device (hereinafter referred to as a "wearer" or "user") are generally referred to as hearing devices. In particular, the present invention relates to hearing devices configured to fully or partially compensate for the hearing loss of a hearing-impaired user. Such hearing devices are also referred to as "hearing aids." Furthermore, there are hearing devices intended to protect or improve the hearing of users with normal hearing, for example, in noisy, stressful, or complex listening situations.

[0003] Hearing devices in general, and hearing aids in particular, are typically designed to be worn on a user's head, particularly in or on the user's ear, and are typically designed as behind-the-ear devices (also referred to as BTE devices, based on the English term "behind the ear") or in-the-ear devices (also referred to as ITE devices, based on the English term "in the ear"). In terms of their internal structure, hearing devices typically include at least one (acoustic-electrical) input transducer, a signal processing unit (signal processor), and an output transducer. During operation of the hearing device, the input transducer or each input transducer receives airborne sound from the hearing device's surroundings and converts this airborne sound into an input audio signal (i.e., an electrical signal conveying information about the surrounding sound). This at least one input audio signal is hereinafter referred to as a "received sound signal." In the signal processing unit, the input audio signal or each input audio signal is processed (i.e., its sound information is modified) to support the user's hearing, particularly to compensate for the user's hearing loss. The signal processing unit outputs the correspondingly processed audio signal (also referred to as an "output audio signal" or "modified sound signal") to an output converter. In other embodiments, the hearing device may also be configured as a handheld device or a desktop device. In this case, the output converter may be formed, for example, by headphones (connected to the handheld device or desktop device via a wired or wireless connection).

[0004] In most cases, the output transducer is designed as an electroacoustic transducer, which converts the (electrical) output audio signal back into airborne sound, where the airborne sound (modified relative to the ambient sound) is output into the user's ear canal. In the case of hearing devices worn behind the ear, the output transducer, also known as the "earpiece" ("receiver"), is typically integrated into the hearing device housing outside the ear. In this case, the sound output by the output transducer is guided into the user's ear canal by means of a sound hose. Alternatively, the output transducer can be arranged in the ear canal, thus outside the housing worn behind the ear. Such hearing devices (from the English term "receiver in canal") are also referred to as RIC devices. Hearing devices worn in the ear (from the English term "completely in canal") are also referred to as CIC devices; these hearing devices are sized so small that they do not protrude from the ear canal.

[0005] In other embodiments, the output transducer can also be designed as an electromechanical transducer, which converts the output audio signal into solid-borne sound (vibrations), wherein the solid-borne sound is output, for example, into the user's skull. Furthermore, there are implantable hearing devices, in particular cochlear implants, and hearing devices whose output transducers directly stimulate the user's auditory nerve.

[0006] The term "hearing system" refers to a single device or a group of devices, including possibly non-physical functional units, that together provide the functions required for the operation of a hearing device. In the simplest case, a hearing system can consist of a single hearing device. Alternatively, a hearing system can include two cooperating hearing devices to supply both ears of a user. In this case, it is referred to as a "binaural hearing system." Additionally or alternatively, a hearing system can include at least one additional electronic device for the or each hearing device, such as a remote control, a charging device, or a programming device. In modern hearing systems, instead of a remote control or dedicated programming device, a control program, particularly in the form of a so-called app, is often provided. This control program is designed to be implemented on an external computer, particularly a smartphone or tablet computer. The external computer itself is typically not part of the hearing system, as it is typically provided separately from the hearing system and not by the hearing system manufacturer. Instead, the external computer, particularly the user's smartphone, is used by the hearing system merely as an external resource for computing power, storage space, and optional communication services.

[0007] Users of hearing devices often have problems understanding (spoken) speech in their surroundings (e.g., speech from a communication partner or speech sounds from a sound reproduction device such as a radio or television). For hearing aid users, this is usually due to the user's hearing impairment, and modern hearing aid technology can often only partially compensate for the user's hearing impairment. For users with normal hearing, supporting or even improving speech understanding through hearing devices is also a complex problem. In both cases, disruptive background noise (especially conversational noise), unclear pronunciation, or pronunciation that is unfamiliar to the user (e.g., with an accent or using a dialect) often increase the difficulty of hearing comprehension. Summary of the Invention

[0008] The technical problem to be solved by the present invention is to effectively support the hearing comprehension of a hearing device user (ie, the user's ability to understand the speech heard).

[0009] According to the present invention, the aforementioned technical problem is solved independently of the present invention in a method for supporting hearing comprehension of a user of a hearing instrument by the features of the present invention. According to the present invention, the aforementioned technical problem is solved independently of the present invention in a hearing system by the features of the present invention. The following description provides advantages of the present invention and some of the embodiments and developments that are considered to be inventive in themselves.

[0010] According to the method, ambient sound containing speech from a user's surroundings is received by means of a hearing device. The speech contained in the received ambient sound is automatically collected and converted into text data. Preferably, the text file is generated in the form of alphanumeric characters in a data technology encoding. However, as an alternative, the collected speech can also be generated into a text file in an alternative form, such as in the form of phonemes, syllables, words, and / or sentence parts (as opposed to alphanumeric characters) in a data technology encoding.

[0011] In a first variant of the method according to the invention, the text data is output as a graphical representation of the text on a screen. Within the scope of the invention, the screen can, in principle, be an integral part of the hearing instrument, for example, when the hearing instrument is designed as a handheld or desktop device. However, it is preferred (particularly in the case of hearing instruments worn on or in the ear) that the text data be output via the screen of a peripheral device connected to the hearing instrument via data transmission technology (e.g., a user's smartphone or smartwatch connected to the hearing instrument).

[0012] In a second variation of the method according to the present invention, text data is converted into synthesized speech and output to the user in the form of an acoustic signal. Preferably, the output occurs via at least one output transducer of a hearing device. When and during the output of the synthesized speech, the original speech sound contained in the ambient sound is preferably attenuated or completely faded out through mechanical and / or signal processing methods. Alternatively, artificially generated ambient noise (such as noise, birdsong, or music) can be mixed into the synthesized speech instead of natural ambient sound to artificially create a naturally occurring noise situation, thereby avoiding potential irritation to the user.

[0013] To effectively support listening comprehension, in both variants of the present invention, the direction of origin and / or at least one speaker characteristic (which characterizes the speaker in more detail) are automatically determined in a time-resolved manner for speech contained in the ambient sound. According to the present invention, the graphical representation of text data derived from the ambient sound or the synthesized speech is modified in a time-resolved manner depending on the determined direction of origin and / or at least one speaker characteristic.

[0014] Here, the direction of incidence of the speech sound contained in the ambient sound relative to the user's head (particularly relative to the user's line of sight) is referred to as the source direction. In other words, the origin of the speech sound contained in the ambient sound is analyzed (preferably using adaptive directional tone reception (adaptive beamforming)). The characteristics of the sounds contained in the ambient sound are generally referred to as speaker characteristics (or also called acoustic features). These characteristics can be used to characterize at least one personal characteristic of the corresponding speaker, and these characteristics can be used to distinguish the speaker from other speakers. For example, at least one speaker characteristic can be selected from timbre, pitch range (i.e., the pitch of the fundamental tone of the sound), speaking speed, or an assumption about the user's gender and / or age (derived from the analysis of the sound).

[0015] The text data or synthesized speech is preferably output to the user in real time, i.e., without a noticeable delay relative to the received ambient sound. When outputting via synthesized speech, the output is preferably performed with a delay of no more than 0.2 seconds, preferably no more than 0.1 seconds. When outputting text data on a screen in the form of graphic characters, the output can be performed with a significant delay relative to the ambient sound without the user perceiving the delay as disruptive, since the text data in the graphic character output can be collected more quickly than spoken speech. In this case, the text data is preferably output with a delay of no more than 0.5 seconds, particularly no more than 0.3 seconds, after receiving the speech sound.

[0016] Two other variations of the method according to the present invention are identical to the first and second variations of the method according to the present invention described above, except that, here, the text data derived by the spoken voice is not output to the user immediately. On the contrary, the text data is recorded (i.e., stored) here for output later. In a third variation of the method according to the present invention, similar to the first variation of the present invention, the text data is recorded by being a graphical representation of text, and this recording is performed in the form of graphic characters. In a fourth variation of the method according to the present invention, similar to the second variation of the present invention, the text data is recorded in the form of an audio signal (i.e., a data signal containing sound information) as a synthesized speech here, and is recorded in the form of sound.

[0017] In the third and fourth variants of the method according to the invention, the source direction and / or at least one speaker characteristic are also automatically determined in a time-resolved manner for speech contained in ambient sounds, wherein the graphical representation of the text data or the synthesized speech is changed in a time-resolved manner depending on the determined source direction and / or at least one speaker characteristic.

[0018] Within the scope of the present invention, the four variations of the present invention described above may be used individually or in any combination. For example, text data derived from ambient sound may be output only as graphic text, only as synthesized speech, or both. Furthermore, within the scope of the present invention, text data may be output only directly to the user, recorded only for later output, or both immediately output and recorded.

[0019] In order to adjust the graphic text output or recording of text data corresponding to the source direction of the speech sound, the display position of the text data on the display screen is preferably changed in a time-resolved manner according to the source direction. For example, when and if the determined source direction shows that the relevant speaker (as viewed in the user's line of sight) is arranged on the left side of the user or in front of the user or on the right side of the user, the text derived from the ambient sound is always displayed in the left area of ​​the display screen, in the middle of the display screen, or in the right area of ​​the display screen. Here, when the source direction of the speech sound changes due to a change of the speaker, movement of the speaker, or movement of the user (particularly head movement), the display position of the text data is changed.

[0020] Alternatively or additionally, a directional symbol associated with the text (e.g., an arrow or a line indicating a speech bubble containing text data) is preferably changed according to the source direction of the speech sound. For example, if the determined source direction indicates that the speaker is located to the left of the user, in front of the user, or to the right of the user, the directional symbol may point to the left, bottom (or top), or right, for example.

[0021] In a particularly intuitive embodiment of the present invention, text data is graphically output or recorded in the form of a virtual reality display (VR display) by inserting the text data into a real image sequence (video) of the user's surroundings recorded during the reception of ambient sound. The text insertion is positionally associated with the image of the relevant sound source in the real image sequence, according to the determined source direction of the speech sound. In other words, the text data containing the speech is inserted into the real image sequence at or near the recorded location from which the speech sound originates in the real surroundings. If the speech sound is produced by a person speaking in the user's surroundings, the text data is displayed near the image of the person in the real image sequence (e.g., in the form of a speech bubble). If the speech sound originates from a sound reproduction device (e.g., a radio or television), the text data is correspondingly displayed near the device. If the source direction changes (e.g., due to a change in the speaker, movement of the speaker, or movement of the user (or the user's image recording device)), the position of the text insertion in the real image sequence is also changed accordingly. That is, in a real image sequence, text data associated with a person or a device is always together with the image of the person or the device.

[0022] In another embodiment of the present invention, the speech of different speakers is visually distinguished from one another by different graphic design of the corresponding associated text data, for example, by selecting the text color, text size, font, and / or background color of the associated text fields. Here, the graphic design is again varied in a time-resolved manner during the output or recording of the text data. For example, text data associated with a first speaker is always displayed in a text field with a red background, while text data associated with a second speaker is always displayed in a text field with a blue background. Within the scope of the present invention, different speakers can be distinguished based on the respective source direction of the speech contained in the ambient sound; for example, a sudden change in the determined source direction is recognized as an indication of a speaker change. However, preferably, different speakers are distinguished based on at least one acquired speaker characteristic (either solely or in addition to evaluating the source direction). Different speakers are identified and distinguished from one another by analyzing the respective acoustics of the speech contained in the ambient sound, for example, based on timbre, pitch register, and / or speaking speed. The graphic design of the text data is modified accordingly to match the respectively identified speaker.

[0023] In order to adjust the output or recording of the synthesized speech in accordance with the source direction of the sound signal, the sound signal or audio signal containing the synthesized speech is preferably generated as a stereo signal with variable surround sound characteristics that always completely or approximately correspond to the surround sound characteristics of the speech sound. In other words, the sound signal or audio signal containing the synthesized speech is generated (particularly by setting the volume, time delay, and / or timbre of the left and right stereo sub-signals identically or differently) so that the synthesized speech appears to the user to come from the source direction determined for the original speech sound. To generate this surround sound characteristic, a head-related transfer function is particularly applied to the synthesized speech.

[0024] Additionally or alternatively, for outputting or recording the synthesized speech, the voice signal or audio signal is preferably modified in a time-resolved manner with respect to timbre and / or pitch register, preferably as a function of at least one speaker characteristic. The synthesized speech is thereby adapted, in particular, to approximately match the characteristics of the original speech sound, and is modified accordingly when the speaker changes. For example, if the original speech in the ambient sound is spoken by a female, male, or child, depending on the voice timbre, the synthesized speech is also generated as a female, male, or child voice.

[0025] Within the scope of the present invention, in principle, if and when the ambient sound contains speech, the speech contained in the ambient sound can always be automatically captured and reproduced and / or recorded as text or sound. Therefore, automatic capture can also be performed when the user speaks. However, these method steps are preferably only applied to speech that does not originate from the user. Therefore, these method steps are preferably not performed if and when the user speaks. This is because these method steps do not provide any benefit for the user's own speech, as the user knows what they are saying.

[0026] The hearing system according to the present invention is generally configured to automatically perform the previously described method according to the present invention in one of the described method variants. Therefore, the previously described embodiments of the method according to the present invention correspond to corresponding embodiments of the hearing system according to the present invention. The aforementioned embodiments of the method according to the present invention and the effects and advantages associated therewith can be correspondingly transferred to the hearing system according to the present invention, and vice versa.

[0027] The hearing system includes at least one hearing device, which in turn includes at least one input converter (preferably multiple input converters), a signal processor, and an output converter. The input converter, or each input converter, is configured to receive ambient sound from a user's surroundings, i.e., to convert the ambient sound into an (input) audio signal, which is then fed to the signal processor. The signal processor is configured to modify the received sound signal. The modification of the received sound signal by the signal processor preferably includes frequency-selective amplification of the received sound signal (in particular, to fully or partially compensate for the user's hearing loss). The (output) audio signal output by the signal processor (which contains the correspondingly modified sound information) can be fed to the output converter for output to the user by the output converter.

[0028] In order to carry out the method according to the invention, the hearing system additionally comprises a speech acquisition unit, an analysis unit and a text pre-processing unit.

[0029] The speech collection unit is configured to automatically convert speech contained in the received ambient sound into text data. The analysis unit is configured to automatically determine the source direction and / or at least one speaker characteristic of the speech contained in the ambient sound in a time-resolved manner. The text preprocessing unit is configured to preprocess the text data for output and / or recording.

[0030] In a variant of the hearing system corresponding to the first or second variant of the method according to the invention, the text preprocessing unit is configured to output the text data to the user as a graphical representation of the text on a screen of the hearing device or a peripheral device connected to the hearing device via data transmission technology, and / or as synthesized speech in the form of an acoustic signal, and in this case, as described in detail according to the method according to the invention, the graphical representation or the synthesized speech is changed in a time-resolved manner as a function of the determined source direction and / or at least one speaker characteristic.

[0031] In another variant of the hearing system corresponding to the third or fourth variant of the method according to the invention, the text preprocessing unit is configured to record the text data as a graphic representation of the text and / or as synthesized speech in the form of an audio signal for subsequent output, and to modify the graphic representation of the text data or the synthesized speech in a time-resolved manner as a function of a determined source direction and / or at least one speaker characteristic.

[0032] Configuring the hearing system to automatically perform the method according to the present invention has programming and / or circuitry. That is, the hearing system according to the present invention includes programming means (software) and / or circuitry (e.g., non-programmable hardware in the form of an ASIC) that automatically performs the method according to the present invention when the hearing system is in operation. The programming means or circuitry for performing the method can be located solely in the hearing device (or devices) of the hearing system. Alternatively, the programming means or circuitry for performing the method can be distributed across one or more hearing devices of the hearing system and at least across another device or software component. For example, the programming means for performing the method can be distributed across at least one hearing device of the hearing system and a control program of the hearing system, wherein the control program of the hearing system is installed on an external electronic device (e.g., a smartphone). As mentioned above, the external electronic device itself is typically not part of the hearing system.

[0033] The hearing device of the hearing system or each hearing device is in particular present in one of the aforementioned configurations (BTE device with internal or external output transducer, ITE device, such as CIC device, hearing implant device, in particular cochlear implant device, hearable device, etc.). In the case of a binaural hearing system, the two hearing devices of the hearing system are preferably designed to be structurally identical.

[0034] The input converter, or each input converter, is in particular an acoustic-electrical converter (i.e., a microphone), which converts airborne sound from the surrounding environment into an electrical input audio signal. The output converter, or each output converter, is preferably designed as an electroacoustic converter (earpiece), which converts the audio signal modified by the signal processing unit back into airborne sound. Alternatively, the output converter is designed for outputting solid-borne sound or for directly stimulating the auditory nerve of the user. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The embodiments of the present invention are described in more detail below with reference to the accompanying drawings. In the accompanying drawings:

[0036] Figure 1 A schematic diagram shows a binaural hearing system consisting of two hearing devices and a control program (operating app), wherein the hearing devices are designed in the form of BTE devices and wherein the control program is installed on a smartphone (which is not part of the hearing system).

[0037] Figure 2A schematic diagram shows a hearing system according to a first embodiment variant, in which, when speech is recognized in the ambient sound received by the hearing device, the recognized speech is automatically converted into text data and output to the user as a graphical representation of text on the screen of a smartphone with the aid of an operating app, wherein the direction of origin of the speech is automatically determined in a time-resolved manner, and wherein the output of the text data is varied depending on the determined direction of origin by correspondingly displaying text sequences with different directions of origin in a text field using indicator arrows whose positions vary depending on the direction of origin.

[0038] Figure 3 Based on Figure 2 The diagram shows a hearing system in a second embodiment variant in which the output of text data is modified as a function of the determined source direction by inserting a text sequence into a real image sequence of the user's surroundings recorded by means of a smartphone in a manner spatially associated with the associated sound source, in particular the associated speaker, and

[0039] Figure 4 Based on Figure 2 The diagram shows a hearing system in a third embodiment variant, in which text data are converted into synthesized speech and output to a user in the form of an acoustic signal via a hearing device, wherein the acoustic signal is generated as a stereo signal with surround sound properties that at least approximately correspond to the surround sound properties of the ambient sound, depending on the source direction.

[0040] In all figures, parts and quantities that correspond to one another are always provided with the same reference symbols. DETAILED DESCRIPTION

[0041] Figure 1 A hearing system 2 is shown which (in general) comprises at least one hearing device 4, in particular a hearing aid configured to support the hearing of a hearing-impaired user. Figure 1 The hearing system 2 shown also includes a second hearing device 4 for supplying the second ear of the user, which second hearing device 4 is designed to be a mirror image of the other hearing device 4 in terms of its internal structure, but is otherwise structurally identical to the other hearing device 4. In the example shown here, the hearing device 4 is a BTE hearing device that can be worn behind the ear of the user. As a further optional part, Figure 1 The shown hearing system 2 further comprises a control program, hereinafter referred to as “Operation App” 6 .

[0042] Each of the two hearing devices 4 includes two microphones 10 as input transducers and an earpiece 12 (receiver) as output transducer within a housing 8. Furthermore, the hearing device 4 or each hearing device 4 includes a battery 14 and signal processing means in the form of a signal processor 16. The signal processor 16 preferably includes a programmable subunit (e.g., a microprocessor) and a non-programmable subunit (e.g., an ASIC).

[0043] The signal processor 16 is supplied with a supply voltage U from the battery 14 .

[0044] During normal operation of the hearing device 4, the microphones 10 receive airborne sound from the surrounding environment of the respective hearing device 4. The microphones 10 convert the sound into an (input) audio signal I containing information about the received sound. The input audio signals I are fed into the hearing device 4 to a signal processor 16, which modifies the input audio signals I to support the user's hearing.

[0045] The signal processor 16 outputs an output audio signal O to the earpiece 12, which output audio signal O contains information about the processed, thus modified, sound.

[0046] The earpiece 12 converts the output sound signal O into a modified airborne sound. The modified airborne sound is transmitted to the user's ear canal through a sound channel 18 and a flexible sound hose (not explicitly shown). The sound channel 18 connects the earpiece 12 to the tip 20 of the housing 8, and the flexible sound hose connects the tip 20 to an earplug inserted into the user's ear canal.

[0047] In accordance with Figure 1 In the example shown, an operating app 6 is installed and operationally available on a user's smartphone 22. The smartphone 22 itself is not part of the hearing system 2. The operating app 6 serves, in particular, as a remote control and programming device for the hearing instruments 4 of the hearing system 2. To this end, during operation of the hearing system 2, the operating app 6 connects to the hearing instruments 4 via a wireless data transmission connection 24, in particular based on the Bluetooth standard, for bidirectional data exchange. To this end, the operating app 6 accesses a transceiver unit (transceiver) of the smartphone 22 (not explicitly shown), which establishes a data transmission connection 24 with a transceiver unit of the corresponding hearing instrument 4 (also not explicitly shown).

[0048] Figure 2The functional components of a first embodiment variant of the hearing system 2 are schematically illustrated in greater detail. Accordingly, the signal processor 16 of each hearing device 4 includes a direction analysis unit (also referred to as a beamformer unit 30 ), which determines the direction of incidence of the respective dominant sound component (and thereby the arrangement of the dominant sound source relative to the user's line of sight, i.e., the frontal direction of the head) by adaptive beamforming or by a plurality of differently oriented beamformers. The beamformer unit 30 can be implemented as an electronic hardware circuit. However, it is preferably implemented as a software component in the signal processor 16 . In addition to the input audio signal I of its own microphone 10 (i.e., the microphone 10 of the hearing device 4 in which the beamformer unit 30 is implemented), the beamformer unit 30 of each hearing device 4 optionally (in a manner not explicitly shown) also considers the input audio signal of the microphone 10 of the respective other hearing device 4 . To this end, the input audio signal I is exchanged wirelessly between the hearing devices 4, if necessary.

[0049] On the one hand, the beamformer unit 30 of each hearing instrument 4 outputs a directional input audio signal I G , the directional input audio signal I G The beamformer unit 30 outputs a source signal R, which indicates the source (incident direction) of the corresponding dominant sound component, for example, in the form of an incident angle relative to the frontal direction of the head in a transverse plane of the head. In the presence of speech activity in the ambient sound, the dominant sound component is usually (at least predominantly) formed by the speech contained in the ambient sound.

[0050] The signal pre-processing unit 32 includes at least one signal processing algorithm, but preferably includes a plurality of signal processing algorithms, using which the directional input audio signal I G The preprocessing is performed to output the audio signal O to support the user's hearing. The signal processing algorithm or algorithms of the signal preprocessing unit 32 are selected, for example, from algorithms for frequency-selective amplification based on the user's audiogram, dynamic compression, active noise suppression, wind noise suppression, feedback suppression, automatic gain control, and the like. These algorithms are preferably implemented as software components.

[0051] The signal processor 16 preferably further includes a signal analysis unit 34. This signal analysis unit 34 includes algorithms that check the received ambient sound (i.e., the input audio signal I) according to predetermined criteria, such as a predetermined noise category, a signal-to-noise ratio, etc., and variably parameterize the algorithm or each algorithm of the beamformer unit 30 and the signal preprocessing unit 32 based on the analysis results. Here, the signal analysis unit 34 includes a speech recognition unit 36, which has algorithms for recognizing speech activity in general (i.e., the presence of speech in the received ambient sound, regardless of the speaker), and, if necessary, the user's own speech in particular.

[0052] Optionally, at least one of the hearing devices 4 further comprises an acceleration sensor 38 .

[0053] In accordance with Figure 2 In the embodiment of FIG, the operating app 6 is configured to convert speech of a speaker other than the user of the hearing system 2 (i.e., different from the user of the hearing system 2) that may be contained in the received ambient sound (and therefore also in the input audio signal I) into text, and to display the text in real time on the screen 40 of the smartphone 22 in the form of subtitles. To this end, the operating app 6 (in the form of software components) includes a speech acquisition unit 42 (also referred to as a speech-to-text converter), an acoustic analysis unit 44, and a text preprocessing unit 46.

[0054] During operation of the hearing system 2, the speech recognition unit 36 ​​continuously checks the input audio signal I to determine whether the received ambient sound contains speech activity that does not originate from the user. The speech recognition unit 36 ​​detects this, for example, by the fact that the speech activity detector of the speech recognition unit 36 ​​provides a positive check result, while the own sound detector of the speech recognition unit 36 ​​does not respond. This check is performed independently of one another in the two hearing devices 4, or alternatively, only by the speech recognition unit 36 ​​of one of the two hearing devices 4. In this case, the speech recognition unit 36 ​​of one of the two hearing devices 4 preferably evaluates the input audio signal I of both hearing devices 4.

[0055] In the aforementioned case where a speech activity not originating from the user himself is recognized in the received ambient sound, the speech recognition unit 36 ​​of at least one of the hearing devices 4 causes the signal processor 16 (here in particular the signal pre-processing unit 32) to transmit the output audio signal O wirelessly via the data transmission connection 24 to the operating app 6. In addition, the operating app 6 receives the source signal R of the beamformer unit 30 and, if present, the acceleration signal B of the acceleration sensor 38 from the at least one hearing device 4.

[0056] The speech acquisition unit 42 converts the speech contained in the output audio signal O into text data T. The acoustic analysis unit 44 determines at least one speaker characteristic S, such as the estimated gender and / or estimated age of the corresponding speaker, based on specific characteristics of the speech sound, particularly the timbre and / or pitch register of the speech sound. Furthermore, the acoustic analysis unit 44 distinguishes different speakers from one another based on the at least one speaker characteristic S by interpreting significant changes in the at least one speaker characteristic S as an indication of a speaker change. To this end, the acoustic analysis unit 44 preferably additionally uses the source signal R and / or the acceleration signal B by inferring a speaker change from sudden changes in the source signal R associated with the speech sound and / or head rotations in response to changes in the direction of the speech sound. The acoustic analysis unit 44 outputs a speaker recognition signal F, along with the determined speaker characteristic S or each speaker characteristic S, to the text preprocessing unit 46, identifying the correspondingly identified speaker.

[0057] In accordance with Figure 2 In the example of , the text pre-processing unit 46 creates a text-containing graphic representation G from the text data T for display on the screen 40 of the smartphone 22, which helps the user of the hearing system 2 to better understand the speech heard from other speakers. In order to provide particularly effective support to the user, the text pre-processing unit 46 graphically displays individual sections of the text data T in different (and therefore user-friendly) forms, and the individual sections of the text data T can be associated with different speakers according to the speaker recognition signal F.

[0058] For example, text preprocessing unit 46 displays the paragraphs of text data T associated with different speakers in different text fields 48. Optionally, text fields 48 are displayed in different colors that characterize the respective speakers. Furthermore, text preprocessing unit 46 optionally selects the respective colors of text fields T based on at least one speaker characteristic S. Thus, for example, text preprocessing unit 46 assigns a red hue to a speaker identified as female based on acoustic analysis, and assigns a blue hue to a speaker identified as male based on acoustic analysis.

[0059] As a further measure for easily assigning the individual text fields 48 to a possible plurality of speakers, the text pre-processing unit 46 arranges a directional symbol for each text field 48 based on the source signal R (here in the form of an arrow 50) so that the position of the symbol on the screen 40 indicates the arrangement of the speaker associated with the text field 48 relative to the user. For text fields 48 associated with a speaker standing to the left of the user, the arrow 50 is displayed in the left area of ​​the text field 48. For text fields 48 associated with a speaker standing to the right of the user, the arrow 50 is displayed in the right area of ​​the text field 48. And for text fields 48 associated with a speaker facing the user directly, the arrow 50 is displayed in the center of the text field 48.

[0060] In addition to or as an alternative to immediately displaying the text field 48 on the screen 40 , the operating app 6 records the graphical representation G in the memory of the smartphone 22 for later output on the smartphone 22 or other device.

[0061] During the time interval in which the user speaks (and the speech recognition unit 36 ​​therefore recognizes the user's own voice in the received ambient sound), the output audio signal O is preferably not forwarded to the operating app 6. In this case, the speech sound is correspondingly not converted into text data T, and the text data T is not graphically displayed on the screen 40.

[0062] Figure 3 An alternative embodiment of a hearing system 2 is schematically shown, in which the graphic representation G of text data T collected from speech sounds is designed as a virtual reality application. Figure 3 The embodiment variant corresponds to Figure 2 But with the implementation plan Figure 2 The implementation plans are different, depending on Figure 3 , a real image sequence V (ie a video of the user's real environment) is recorded with the help of the camera 52 of the smartphone 22 and displayed on the screen 40 of the smartphone 22 in real time (preferably with a maximum delay of 0.5 seconds).

[0063] For the graphical representation of text data T collected from speech sound, Figure 3In the example shown in FIG, text preprocessing unit 46 embeds a text paragraph associated with each speaker (different from the user) into a speech bubble 54. Here, text preprocessing unit 46 inserts speech bubble 54 into real image sequence V near the image of the associated speaker or sound-generating object (radio, television, etc.) according to source signal R. Text preprocessing unit 46 arranges indicator lines 56 of speech bubble 54 so that indicator lines 56 point toward the speaker or sound-generating object. To better associate speech bubble 54 with the corresponding associated sound source, operating app 6 preferably uses an image recognition unit 58 connected downstream of camera 52. This image recognition unit 58 automatically recognizes, for example, the heads of captured persons or other sound sources in the real image sequence. Operating app 6 preferably uses a standard function of the camera app of smartphone 22 as image recognition unit 58.

[0064] In accordance with Figure 3 In an embodiment variant of the hearing system 2, in addition to or as an alternative to the immediate display on the screen 40, the operating app 6 can also record a graphic representation G of the text data T (here a real image sequence V with inserted speech bubbles 54) in the memory of the smartphone 22 for later output on the smartphone 22 or another device.

[0065] Figure 4 A further embodiment variant of the hearing system 2 is shown, which, apart from the differences described below, corresponds to the embodiment variant according to Figure 2 and Figure 3 Implementation variant. Figure 2 and Figure 3 The implementation variants are different, according to Figure 4 In an embodiment variant, instead of graphic pre-processing of the text data T, acoustic pre-processing of the text data T in the form of synthesized speech L is provided.

[0066] To this end, according to Figure 4In a variant embodiment, the text preprocessing unit 46 includes a speech synthesis unit 60 and an ambient sound generation unit 62. The speech synthesis unit 60 converts the text data T into synthesized speech L. Based on the determined at least one speaker characteristic S and the speaker recognition signal F, the speech synthesis unit 60 generates the synthesized speech L for each of the identified different speakers using correspondingly different assigned artificial voices. In particular, the speech synthesis unit 60 generates the synthesized speech L for each of the identified different speakers using a voice that is similar to the speaker's actual voice according to the at least one speaker characteristic F. Thus, for example, a text passage associated with a (female) speaker is synthesized using a female-sounding voice, while a text passage associated with a (male) speaker is synthesized using a male-sounding voice.

[0067] Furthermore, each text passage may optionally be synthesized using a voice that matches the estimated age of the corresponding speaker. Thus, for example, a text passage associated with a young speaker may be synthesized using a young-sounding voice, while a text passage associated with an older speaker may be synthesized using an older-sounding voice.

[0068] In all cases, it is preferred to generate the synthesized speech L as a clear-enunciated standard speech. Therefore, when generating the synthesized speech L, it is preferred to avoid using accents, dialects, and / or unclear pronunciations.

[0069] The speech synthesis unit 60 outputs the synthesized speech L generated in this manner as an audio signal to the downstream surround sound generation unit 62. Based on the source signal R, the surround sound generation unit 62 converts the synthesized speech L into a stereo signal ST having surround sound characteristics that fully or approximately correspond to those of the ambient sound. The operating app 6 feeds this stereo signal ST back to the hearing device 4 (specifically, the signal preprocessing unit 32). The hearing device 4 outputs this stereo signal ST (preferably in real time, i.e., with a maximum delay of 0.2 seconds) through the earpieces 12 of the hearing device 4 as airborne sound to the user's two ears.

[0070] Due to the surround sound characteristics of the stereo signal ST generated by the surround sound generation unit 62, a text passage of a speaker located to the left, right, or front of the user sounds as if the synthesized speech L will also come from the left, right, or front from the user's perspective. To perform this conversion, the surround sound generation unit 62 applies a stored head-related transfer function to the synthesized speech L.

[0071] While outputting the synthesized speech L to the user, the signal preprocessing unit 32 of the hearing device 4 preferably fades out the received real ambient sounds, in particular (and preferably selectively fades out only or to a greater extent) the ambient sounds in the frequency range relevant to the received speech, to prevent the synthesized speech L in the output audio signal O from interferingly overlapping with the received real speech. Artificially generated ambient noise (noise, bird song, music) is optionally mixed into the synthesized speech L to avoid unnaturally appearing noise situations. Therefore, it is particularly preferred to synthesize a complete, naturally appearing noise background.

[0072] In this embodiment variant of the hearing system 2, in addition to or instead of immediately outputting the pre-processed text data T, here the synthesized speech L, the operating app 6 can also record the synthesized speech L for later output on the smartphone 22 or another device. Figure 4 Implementation variants and based on Figure 2 and Figure 3 , in that the operating app 6 optionally causes a simultaneous output and / or recording of a graphical representation G of the text data T and of the synthesized speech L.

[0073] The present invention is particularly apparent from the above-described embodiments, but the present invention is by no means limited to these embodiments. On the contrary, further embodiments of the present invention can be derived from the above description.

[0074] Reference Signs List

[0075] 2 Hearing System

[0076] 4 Hearing devices

[0077] 6. Operate the App

[0078] 8 Housing

[0079] 10 microphones

[0080] 12 Receiver

[0081] 14 Batteries

[0082] 16 Signal Processor

[0083] 18 sound channels

[0084] 20 Tip

[0085] 22 Smartphone

[0086] 24 data transmission connections

[0087] 30 beamformer units

[0088] 32 signal preprocessing unit

[0089] 34 Signal Analysis Unit

[0090] 36 speech recognition units

[0091] 38 accelerometer

[0092] 40 screens

[0093] 42 Voice Collection Unit

[0094] 44 Sound Analysis Unit

[0095] 46 Text Preprocessing Unit

[0096] 48 Text Fields

[0097] 50 arrows

[0098] 52 cameras

[0099] 54 speech bubbles

[0100] 56 indicator line

[0101] 58 Image Recognition Unit

[0102] 60 speech synthesis units

[0103] 62 surround sound generation units

[0104] B acceleration signal

[0105] F Speaker recognition signal

[0106] G (Graph)

[0107] I (input) audio signal

[0108] I G (directional input) audio signal

[0109] L (synthesized) speech

[0110] O (output) audio signal

[0111] R Source Signal

[0112] S Speaker characteristics

[0113] ST stereo signal

[0114] T Text data

[0115] U supply voltage

[0116] V Real image sequence

Claims

1. A method for supporting hearing comprehension of a user of a hearing device (4), -in, receiving ambient sound including speech from the user's surroundings by means of the hearing device (4), - wherein the speech contained in the received ambient sound is automatically converted into text data (T), and - wherein the text data (T) - as a graphic representation (G) of text on a screen (40) of the hearing device (4) or of a peripheral device (22) connected to the hearing device via data transmission technology, and / or -- as synthesized speech (L) in the form of an acoustic signal, Output to the user, Wherein, for the speech contained in the environmental sound, the source direction (R) and / or at least one speaker characteristic (S) are automatically determined in a time-resolved manner, and wherein, based on the determined source direction (R) and / or at least one speaker characteristic (S), the graphical representation (G) of the text data (T) or the synthesized speech (L) is changed in a time-resolved manner.

2. The method according to claim 1, in, The text data (T) or the synthesized speech (L) is output to the user in real time.

3. A method for supporting hearing comprehension of a user of a hearing device (4), -in, receiving ambient sound including speech from the user's surroundings by means of the hearing device (4), - wherein the speech contained in the received ambient sound is automatically converted into text data (T), and - wherein the text data (T) -- as a graphical representation of text (G), and / or -- as synthesized speech (L) in the audio signal, Recording is performed for later output, wherein, for the speech contained in the ambient sound, a source direction (R) and / or at least one speaker characteristic (S) is automatically determined in a time-resolved manner, and wherein, depending on the determined source direction (R) and / or at least one speaker characteristic (S), a graphical representation (G) of the text data (T) or the synthesized speech (L) is changed in a time-resolved manner.

4. The method according to any one of claims 1 to 3, in, For a graphical representation (G) of the text data (T), the display position and / or a directional symbol (50, 56) associated with the text is changed in a time-resolved manner depending on the determined source direction (R).

5. The method according to any one of claims 1 to 4, in, The graphical representation (G) of the text data (T) is realized in the form of an insertion of the text into a sequence of real images (V) of the user's surroundings recorded during the reception of the ambient sound, and wherein the insertion of the text is positionally associated with the image of the relevant sound source in the sequence of real images (V).

6. The method according to any one of claims 1 to 4, in, For the graphical representation (G) of the text data (T), the text color, text size, font and / or background color of the associated text field (48) are changed in a time-resolved manner.

7. The method according to any one of claims 1 to 6, in, The synthesized speech (L) is generated as a stereo signal (ST) having surround sound characteristics that completely correspond to or approximately correspond to surround sound characteristics of the ambient sound according to the source direction (R).

8. The method according to any one of claims 1 to 7, in, The synthesized speech (L) is modified in a time-resolved manner with respect to timbre and / or pitch register as a function of at least one speaker characteristic (S).

9. A hearing system (2) having at least one hearing device (4), the hearing device having at least one input transducer (10) for receiving ambient sound from a user's surroundings, a signal processor (16) for modifying the received sound signal, and an output transducer (12) for outputting the modified sound signal to the user, -in, The hearing system (2) additionally comprises a speech acquisition unit (42), an analysis unit (30, 44) and a text preprocessing unit (46). - wherein the voice collection unit (42) is configured to automatically convert the voice contained in the received environmental sound into text data (T), wherein the analysis unit (30, 44) is configured to automatically determine a source direction (R) and / or at least one speaker characteristic (S) in a time-resolved manner for speech contained in the ambient sound, and - wherein the text preprocessing unit (46) is configured to convert the text data (T) - as a graphic representation (G) of text on a screen (40) of the hearing device (4) or of a peripheral device (22) connected to the hearing device via data transmission technology, and / or -- as synthesized speech (L) in the form of an acoustic signal, The graphical representation (G) or the synthesized speech (L) is output to a user and is changed in a time-resolved manner depending on the determined source direction (R) and / or at least one speaker characteristic (S).

10. A hearing system (2) having at least one hearing device (4), the hearing device having at least one input transducer (10) for receiving ambient sound from a user's surroundings, a signal processor (16) for modifying the received sound signal, and an output transducer (12) for outputting the modified sound signal to the user, -in, The hearing system (2) additionally comprises a speech acquisition unit (42), an analysis unit (30, 44) and a text preprocessing unit (46). - wherein the voice collection unit (42) is configured to automatically convert the voice contained in the received environmental sound into text data (T), wherein the analysis unit (30, 44) is configured to automatically determine a source direction (R) and / or at least one speaker characteristic (S) in a time-resolved manner for speech contained in the ambient sound, and - wherein the text preprocessing unit (46) is configured to convert the text data (T) -- as a graphical representation of text (G), and / or -- as synthesized speech (L) in the form of an audio signal, A recording is performed for later output and the graphical representation (G) or the synthesized speech (L) is modified in a time-resolved manner as a function of a determined source direction (R) and / or at least one speaker characteristic (S).

Images