Method and device for speech understanding ability training

Abstract

A method for performing speech understanding ability training and a corresponding apparatus are presented. Sound from a participant's environment is recorded. A speech of a speaker different from the participant is extracted from the recorded sound and at least one characteristic speech attribute and / or speech attribute of the speaker is determined from the extracted speech. A plurality of test audio sequences are created, where each of the plurality of test audio sequences contains a synthetic speech of phonemes or a combination of phonemes, and the speech is synthesized to conform to at least one characteristic speech and / or speech attribute of the speaker. A training step is performed in which one of the plurality of test audio sequences is selected, converted into sound and output to the participant. A response of the participant is collected indicating the phonemes or combinations of phonemes understood by the participant, and feedback is output to the participant indicating whether the phonemes or combinations of phonemes understood by the participant correspond to the phonemes or combinations of phonemes output to the participant in the training step.

Description

TECHNICAL FIELD

[0001] The present invention relates to a method and an apparatus for conducting speech understanding ability training, in particular for a user of a hearing device. BACKGROUND

[0002] Generally, a hearing device is an electronic device which is designed to support the hearing of a person wearing the electronic device, referred to as the "user" or "wearer" of the hearing device. In particular, the present invention relates to hearing devices which are specifically configured to at least partially compensate for a hearing impairment of a user. Such hearing devices are also referred to as "hearing aids". In addition to such hearing aids, there are also hearing devices which are designed to support a user with normal hearing, i.e. a person without a hearing impairment. Such hearing devices, sometimes referred to as "Personal Sound Amplification Products" (PSAP), can be provided, for example, to enhance the hearing of the wearer in complex acoustic environments or to protect the hearing of the wearer from damage or overstrain.

[0003] Hearing devices, in particular hearing aids, are typically designed to be worn in or on the ear of a user, for example as a Behind-The-Ear (BTE) or In-The-Ear (ITE) device. In terms of its internal structure, a hearing device typically comprises an (acoustic-electric) input transducer, a signal processor and an output transducer. During operation of the hearing device, the input transducer picks up sound signals from the environment of the hearing device and converts them into an input audio signal, i.e. an electrical signal conveying the sound information. The input audio signal is processed in the signal processor, in particular amplified according to frequency, for example to compensate for a hearing impairment of the user. The signal processor outputs the processed signal, also referred to as output audio signal, to the output transducer. Most often, the output transducer is an electro-acoustic transducer, also referred to as "receiver", which converts the output audio signal into a processed airborne sound which is emitted into the ear canal of the user. Alternatively, the output transducer can be an electromechanical transducer which converts the output audio signal into a structure-borne sound (vibration) which is transmitted to the skull of the user, for example.

[0004] In addition to the typical hearing aids, there are also implantable hearing aids such as cochlear implants and hearing devices whose output transducer directly stimulates the auditory nerve of the user.

[0005] The term "hearing system" refers to one device or an assembly of devices and / or other structures that provide the functionality required for the operation of a hearing device. A hearing system can consist of a single, stand-alone hearing device. Alternatively, a hearing system can comprise a hearing device and at least one further electronic device, which can be, for example, one of another hearing device for the other ear of the user, a remote control and a programming tool for the hearing device. Furthermore, modern hearing systems often comprise a hearing device and a software application for controlling and / or programming the hearing device, which is referred to as "hearing app" in the following, installed on or installable on a computer, in particular a mobile communication device such as a mobile phone (smartphone). In the latter case, the computer is usually not part of the hearing system but only used by the hearing system as a resource for data storage, digital capabilities and communication services. Most of the time, the computer, in particular the mobile communication device, on which the hearing app is installed or installable, will be manufactured and sold independently of the hearing system.

[0006] A serious problem faced by persons starting to use a hearing device is the speech understanding ability, i.e. the ability of the person to understand speech. On the one hand, part of these problems is caused by the fact that any hearing device will change or interfere with natural sound to some extent as a result of the attenuation and / or signal processing performed by the hearing device. Thus, although the hearing device can amplify the sound provided to the user compared to the original environmental sound, information typically used by the human hearing, such as spectral cues or rapid amplitude changes or slight binaural amplitude differences or delays of environmental sound, can be lost. On the other hand, a hearing impaired person starting to use a hearing aid can have experienced a more or less prolonged period of unaided hearing loss during which the brain of the person can have lost the speech understanding ability. For the reasons mentioned above, both hearing impaired persons and persons with normal hearing usually have to train the speech understanding ability when starting to use a hearing device. A traditional method of training the speech understanding ability often involves playing pre-recorded audio sequences containing phonemes or combinations of phonemes to the user. These combinations of phonemes can correspondingly include a vowel to be understood embedded in a consonant environment (e.g. "mom", "mum", "mem",...) or a consonant to be understood embedded in a vowel environment (e.g. "akka", "alla", "atta",...).

[0007] However, the pre-recording of audio material for such training is both time consuming and expensive. Furthermore, it is difficult to create or select audio material suitable for the individual user's requirements to be used in speech understanding ability training, as the requirements of different hearing device users vary greatly. As a result, hearing device users participating in traditional speech understanding ability training are often provided with a too small set of test audio sequences and / or unsuitable audio test sequences that do not allow for a robust benefit in real-life noisy environments. Thus, for many hearing device users, traditional speech understanding ability training is a tedious and slow process without any hope of significant success. SUMMARY

[0008] It is therefore an object of the present invention to provide a method for conducting speech understanding ability training that should be time efficient (in terms of time required for achieving significant training success), easy to implement and / or affordable for most individuals. It is another object of the present invention to provide an efficient device suitable for performing the method.

[0009] According to the present invention, the above mentioned technical problems are solved by a method for conducting speech understanding ability training according to the present invention and a device for conducting speech understanding ability training according to the present invention. Preferred embodiments of the present invention are described in the following description.

[0010] According to the method for conducting speech understanding ability training, sound from an environment of a participant is recorded. Speech of a (first) speaker different from the participant is extracted from the recorded sound and at least one characteristic voice property and / or speech property of the (first) speaker is determined from the extracted speech. A plurality of (first) test audio sequences (also referred to as "phoneme stimuli") is created, wherein each of the plurality of (first) test audio sequences comprises synthesized speech of a phoneme or a combination of phonemes. According to the present invention, the speech is synthesized to comply with the at least one characteristic voice property and / or speech property of the (first) speaker;

[0011] Here and in the following, speech understanding ability training denotes a training applied to a participant, in particular a user of a hearing device, that aims at improving the participant's ability to understand speech.

[0012] In a (first) training step of the method,

[0013] - one of the test audio sequences of the plurality of (first) test audio sequences is selected, converted to sound and output to the participant;

[0014] - a response of the participant indicating the phoneme or the combination of phonemes understood by the participant is collected; and

[0015] - outputting feedback to the participant, wherein the feedback informs the participant whether he or she has indicated the correct phoneme or phoneme combination or not (i.e. whether the phoneme or phoneme combination the participant indicated to have understood corresponds to the phoneme or phoneme combination actually output to the participant in that training step).

[0016] Here and in the following, the following definitions are used:

[0017] - The term "sound" generally refers to a signal directly causing an auditory perception of a client. In particular, sound can be transmitted by pressure oscillations in air (airborne sound) or vibrations in a liquid or solid structure (structure-borne sound) or an electrical signal directly stimulating the auditory nerves of a participant.

[0018] - In contrast to "sound", the term "audio signal" denotes an electrical signal transmitting sound information and being converted into sound when fed to an output transducer.

[0019] - The term "speech" denotes a spoken text, which can be spoken by a natural person or can be a synthetic representation of a spoken text (synthetic speech).

[0020] - In contrast to "speech", the term "voice" denotes a sound produced by a person's vocal cords or a synthetic representation of such a sound, wherein "voice" generally will transmit "speech", but can also transmit non-textual information (e.g. humming).

[0021] - A "participant" is a person applying the method for conducting speech understanding ability training. Preferably, the participant is a user of a hearing device and, if the method is performed using a hearing system, the participant is identical to the user of the hearing system. However, in some embodiments of the present application, the method can be applied to a person not using a hearing device.

[0022] - A "speaker different from the participant" or an "arbitrary speaker different from the participant" is also referred to as "communication partner". The terms such as "first" or "second" speaker / communication partner are only used for the purpose of uniquely marking the respective speaker / communication partner; in particular, the terms "first" or "second" are not intended to designate an order of speaker interaction. Thus, in general, the first or second speaker / communication partner can be any person communicating with the participant.

[0023] At least one "speech attribute" is a measurable quantity that characterizes the speech of the corresponding communication partner. For example, the at least one speech attribute can be selected from the group consisting of pitch (fundamental frequency), harmonic structure, and (secondarily) intensity. At least one "speech attribute" is a measurable quantity that characterizes the speech of the corresponding communication partner without taking into account speech characteristics. For example, the at least one speech attribute can be selected from the group consisting of speaking rate (e.g., average syllables per second), intonation (i.e., speech melody), and pronunciation.

[0024] Feedback that informs the participant that he or she has indicated a correct phoneme or phoneme combination is called "positive feedback," whereas feedback that informs the participant that he or she has indicated an incorrect phoneme or phoneme combination (i.e., a phoneme or phoneme combination that is different from the phoneme or phoneme combination actually output in the (first) training step) is called "negative feedback."

[0025] The present invention is based on the understanding that participants are most familiar with the voice and speech characteristics of the people they interact with on a daily basis. Therefore, since participants generally understand these known voices better than unknown voices, the likelihood that phoneme stimuli created from known voices will be appropriate for the respective participant is very high. Therefore, presenting phoneme stimuli with known voice and / or speech characteristics is a very promising method for quickly achieving training success.

[0026] In particular, hearing device users benefit from being presented with phoneme stimuli that have known speech and voice characteristics, because even if these known characteristics are altered by the hearing device's signal processing, the hearing device user can still recognize them relatively easily. Another significant advantage of the present invention is that collecting speech and voice characteristics from a participant's environment and synthesizing phoneme stimuli based on these characteristics is easy to implement and involves minimal effort and financial resources. Thus, the present invention provides the possibility of collecting high-quality training material for speech comprehension training tailored to each individual participant in a time-saving, highly cost-effective (low-cost) manner that can be used by a large number of individual users.

[0027] In general, the feedback provided to the participant at the end of the (first) training step can be provided in any way perceptible to the user, e.g. as a text message or picture on a display screen or as an acoustic message. However, at least in case of negative feedback, i.e. if the participant indicates a phoneme or phoneme combination understood not to correspond to the phoneme or phoneme combination output to the participant, it is preferred that the feedback comprises an acoustic message audible to the participant; here, the feedback comprises the speech sound of the phoneme or phoneme combination indicated to be understood by the participant and a repetition of the speech sound of the phoneme or phoneme combination output to the participant in the first training step. That is, the phoneme or phoneme combination considered to be understood by the participant is repeated and accompanied by the phoneme or phoneme combination provided to the participant during the first training step; e.g. the feedback is provided as a synthesized speech sound comprising the sentence "Your choice was incorrect. You chose "agga", but you heard "akka". Here, the speech sound of the phoneme or phoneme combination indicated to be understood by the participant is synthesized to conform to the at least one characteristic voice and / or speech property of the first speaker. In other words, to simplify the comparison of the sounds of the two phonemes or phoneme combinations, the indicated phoneme or phoneme combination is synthesized with the same voice and / or speech characteristics as the phoneme or phoneme combination provided to the participant during the first training step.

[0028] In a preferred embodiment of the application, to increase the variety of training material, speech of a (at least) second speaker different from the participant is extracted from the recorded sound, at least one characteristic voice and / or speech property of the second speaker is determined from the extracted speech, and a plurality of second test audio sequences is created, wherein, as in the case of the plurality of first test audio sequences, each of the plurality of second test audio sequences comprises a synthesized speech of a phoneme or phoneme combination. However, unlike the plurality of first test audio sequences, the speech of the plurality of second test audio sequences is synthesized to conform to the at least one characteristic voice and / or speech property of the second speaker.

[0029] According to an embodiment of the present application, in particular in case the participant fails to successfully pass the training step, the training mode is changed using a plurality of test audio sequences synthesized to conform to the speech and / or phonetic characteristics of a different communication partner of the participant. In this case, i.e. if in the first training step the participant indicates an understood phoneme or phoneme combination which does not correspond to the phoneme or phoneme combination output to the participant, in a second training step a test audio sequence of the plurality of second test audio sequences is selected, converted to sound and output to the participant, wherein the selected test audio sequence of the plurality of second test audio sequences contains the same phoneme or phoneme combination as the test audio sequence output in the first training step. I.e. in case of a failure of the first training step, in the second training step the same phoneme or phoneme combination is presented again in a speech similar to the speech of a second speaker. In other respects, the second training step is similar to the first training step in that the participant's response indicating the phoneme or phoneme combination understood by the participant is collected and feedback is output to the participant whether the phoneme or phoneme combination indicated to be understood by the participant corresponds to the phoneme or phoneme combination output to the participant in the second training step.

[0030] In order to select the first speaker (and, if applicable, the second speaker) in an appropriate manner, the speech of a plurality of communication partners (preferably more than two speakers different from the participant) is preferably extracted from the recorded sound. The extracted speech is evaluated, e.g. using statistical methods, with respect to the frequency and / or time period of speech of each of the speakers. Here, one of the plurality of speakers who speaks most frequently or longest is selected as the first speaker. If applicable, another of the plurality of speakers who speaks most frequently or longest is preferably selected as the second speaker.

[0031] Here, in an advantageous embodiment of the present application, in contrast to what might seem straightforward, the speaker who speaks most frequently or longest is not selected as the first speaker, but as the second speaker. Instead, the speaker who speaks second most frequently or second longest is selected as the first speaker. Thus, since speech understanding of the most familiar speech is easiest for the participant, the preferred training arrangement first starts with a less easy training step by presenting the phoneme stimuli in a less familiar speech and, if the first training step fails, reduces the difficulty in the second training step by repeating the phoneme stimuli with the most familiar speech. Thereby, the training efficiency is improved.

[0032] Optionally, the recorded sound from the participant's environment is de-noised before determining the respective speaker's at least one characteristic speech property and / or speech property from the extracted speech. In a preferred embodiment of the present invention, de-noising (e.g. by active noise cancellation and / or beamforming) is combined with scene classification (i.e. classifying a pre-defined acoustic environment such as "speech in quiet environment", "speech in noisy environment", etc.). Scene classification is used to determine the degree of de-noising required to enable the determination of at least one characteristic speech property and / or speech property of satisfactory quality. The goal is to know beforehand how suitable the participant's acoustic environment is for the intended "voice cloning" (reconstruction of a selected voice from the environment), the best option being a "speech in quiet environment" environment. In less preferred environments, de-noising can be applied. According to the present invention, scene classification can also be used to identify a current acoustic environment that is not suitable for determining at least one characteristic speech property and / or speech property of a communication partner. In these cases, the determination of the communication partner's speech and / or speech property can be stopped or rejected, or the participant can be prompted to change the acoustic environment.

[0033] Preferably, artificial intelligence (AI), e.g. a (deep) neural network, is used to create the test audio sequence, in particular the plurality of first test audio sequences (and, where applicable, the plurality of second test audio sequences). Suitable AI models that can be used for this purpose according to the present invention are WaveNet (see A. van den Oord et al., "WaveNet: A generative model for raw audio", 2016, https: / / arxiv.org / pdf / 1609.03499.pdf), Tacotron (Y. Wang, "Tacotron: Towards End-to-End Speech Synthesis", 2017, https: / / arxiv.org / pdf / 1703.10135.pdf) and variational autoencoder (VAE).

[0034] In a particularly preferred embodiment of the application, a hearing device worn on or in the ear of the participant is used to record the sound from the participant's environment and to output the test audio sequence or each test audio sequence to the participant. Furthermore, the speech of the speaker or each speaker different from the participant is preferably extracted from the recorded sound using the hearing device. The use of a hearing device makes it very easy to implement the application, since most of the technical functionality required to perform the method, such as a means for recording the sound from the user's environment, a means for outputting sound to the user, a signal processor operable to identify speech activity and the user's own speech (and thus to identify the speech of speakers different from the user), a scene classifier, a signal processor operable to denoise the recorded sound, etc., are readily available in modern hearing devices. On the other hand, the creation of the test audio sequence is preferably carried out by a remote computing service, in particular a cloud service, with which the hearing device is connected, directly or indirectly, for example via the user's mobile phone, for data exchange.

[0035] In a further embodiment of the method according to the application, at least one speaker different from the participant is prompted to pronounce a plurality of predefined phonemes or phoneme combinations. The phonemes or phoneme combinations pronounced by the at least one speaker are recorded and stored as a plurality of test audio sequences. Each of the plurality of test audio sequences contains the respective phoneme or phoneme combination. A training step is carried out in which one of the plurality of test audio sequences is selected, converted into sound and output to the participant. The participant's response indicating the phoneme or phoneme combination understood by the participant is collected. Feedback is output to the participant as to whether the phoneme or phoneme combination indicated as understood by the participant corresponds to the phoneme or phoneme combination output to the participant in the training step.

[0036] As mentioned above, a further embodiment of the application is a device for carrying out a speech understanding ability training, for example as defined above. A particular embodiment of the device according to the application is a hearing system comprising a hearing device, in particular a hearing aid, which can be implemented in any of the embodiments described in the introductory part of the specification, in particular as a BTE device or an ITE device. The hearing system can also comprise a mobile device or a software application installed on a mobile device, in particular a smartphone (hearing app). In the latter case, the mobile device itself is preferably not part of the hearing system, but is manufactured and marketed independently thereof.

[0037] Generally, the device according to the application, in particular the hearing system, is configured for carrying out the method according to the application. Furthermore, any embodiment of the method has a corresponding embodiment of the device. Thus, all explanations and explanations regarding the changes, advantages and effects of the different embodiments of the method apply identically and can be transferred to the corresponding embodiments of the device and vice versa.

[0038] In particular, the device comprises

[0039] - an input transducer for recording sound from the environment of the participant,

[0040] - a signal analysis unit designed to extract speech from the recorded sound of a first speaker different from the participant, to determine at least one characteristic voice and / or speech property of the first speaker from the extracted speech, and

[0041] - a speech synthesis unit designed to create a plurality of first test audio sequences (phoneme stimuli), wherein each of the plurality of first test audio sequences contains a synthesized speech of a phoneme or a combination of phonemes and the speech is synthesized to comply with the at least one characteristic voice and / or speech property of the first speaker.

[0042] The device further comprises a training unit designed to carry out a first training step, in which

[0043] - one of the plurality of first test audio sequences is selected, converted into sound and output to the participant,

[0044] - the participant's response indicating the phoneme or the combination of phonemes understood by the participant is collected, and

[0045] - a feedback is created by the participant indicating whether the phoneme or the combination of phonemes understood corresponds to the phoneme or the combination of phonemes output to the participant in the first training step, which is output to the participant via an output transducer of the hearing device.

[0046] If the device is formed by a hearing system comprising a hearing device and a hearing app, then preferably the input transducer, the signal analysis unit and the output transducer are implemented in the hearing device, while the training unit is implemented as part of the hearing app. According to the application, the speech synthesis unit can also be implemented as part of the hearing app (or as part of the hearing device). However, it is preferred that the speech synthesis unit is implemented as a remote service (for example in a data cloud), to which the hearing device (or the hearing system) is connected for data transmission. In a further embodiment, at least part of the signal analysis unit and the speech synthesis unit can be integrated in one unit, which can be an AI model, for example.

[0047] In an embodiment of the application, the training unit is designed to, if in the first training step the phoneme or the combination of phonemes indicated to be understood by the participant does not correspond to the phoneme or the combination of phonemes output to the participant, provide feedback such that the feedback contains the speech sound of the phoneme or the combination of phonemes indicated to be understood by the participant and a repetition of the speech sound of the phoneme or the combination of phonemes output to the participant in the first training step. Here, the speech sound of the phoneme or the combination of phonemes indicated to be understood by the participant is synthesized by the speech synthesis unit to conform to the at least one characteristic voice property and / or speech property of the first speaker.

[0048] In another embodiment of the application, the signal analysis unit is designed to extract the speech of the second communication partner from the recorded sound and to determine the at least one characteristic voice property and / or speech property of the second communication partner from the extracted speech. Here, the speech synthesis unit is designed to create a plurality of second test audio sequences, wherein each of the plurality of second test audio sequences contains a synthesized speech of a phoneme or a combination of phonemes and the speech is synthesized to conform to the at least one characteristic voice property and / or speech property of the second communication partner. The training unit is designed to perform a second training step as described above.

[0049] Preferably, the speech analysis unit is designed to extract the speech of a plurality of communication partners from the recorded sound, to evaluate the extracted speech with respect to the frequency and / or the time period at which each of the speakers spoke, and to select one of the plurality of speakers who spoke most frequently or longest in time as the first (or second) speaker as described above.

[0050] Preferably, the device (in particular the hearing system) further comprises a signal processor (which can be implemented, for example, as part of a hearing device) which is designed to denoise the recorded sound before determining the at least one characteristic voice property and / or speech property of the respective speaker from the extracted speech.

[0051] Features of all embodiments of the method and all embodiments of the device according to the application can be combined with one another, unless stated otherwise. BRIEF DESCRIPTION OF DRAWINGS

[0052] Embodiments of the application will be described in more detail below with reference to the attached drawings, in which:

[0053] Figure 1A hearing system is shown, which comprises a hearing device to be worn on the ear of a user and a software application (hearing app) installed on a mobile phone, wherein the method for training the speech understanding ability of a user according to the present application is implemented as a function of the hearing app; and

[0054] Figures 2 to 4 A block diagram of different embodiments of the hearing system of Figure 1

[0055] In the drawings, like reference numerals will be used to indicate like parts, structures and elements unless otherwise specified. DETAILED DESCRIPTION

[0056] Figure 1 A hearing system 2 is shown, which comprises a hearing device 4 configured for being worn in or on one of the ears of a user. Preferably, the hearing device 4 is a hearing aid, i.e. a hearing device configured for supporting the hearing of a hearing impaired user. As Figure 1 is shown, the hearing device 4 can be designed as a behind-the-ear (BTE) hearing device, for example. Optionally, the system 2 comprises a second hearing device (not shown) to be worn in or on the other ear of the user for providing binaural support to the user.

[0057] The hearing device 4 comprises two microphones 8 as input transducers and a receiver 10 as output transducer inside a housing 6. The hearing device 4 further comprises a battery 12 and a signal processor 14. Preferably, the signal processor 14 comprises both programmable sub-units (e.g. microprocessors) and non-programmable sub-units (e.g. ASICs).

[0058] The signal processor 14 is powered by the battery 12, i.e. the battery 12 provides a supply voltage U to the signal processor 14.

[0059] ​During normal operation of the hearing device 4, the microphone 8 picks up an air-borne sound signal from the environment of the hearing device 4. The microphone 8 converts the air-borne sound into an (raw) input audio signal I (also referred to as "picked-up sound signal"), i.e. an electrical signal containing information of the picked-up sound. The input audio signal I is fed to the signal processor 14. The signal processor 14 processes the input audio signal I in order to, for example, provide directional sound information (beamforming), perform noise reduction and dynamic compression, and individually amplify different spectral parts of the input audio signal I based on the user's hearing map data to compensate for user-specific hearing loss. The signal processor 14 issues an output audio signal O (also referred to as "processed sound signal"), i.e. an electrical signal containing information of the processed sound, to the receiver 10. The receiver 10 converts the output audio signal O into a processed air-borne sound, which is emitted through a sound channel 16 connecting the receiver 10 to a tip 18 of the housing 6 and a flexible sound tube (not shown) connecting the tip 18 to an ear plug inserted into the user's ear canal.

[0060] In addition to the hearing device 4, the hearing system 2 comprises a software application (referred to as "hearing app" 20 in the following) installed on the user's mobile phone 22. Here, the mobile phone 22 is not part of the hearing system 2. Rather, it is only used by the hearing system 2 as an external resource providing computing power, data storage (memory) and communication services.

[0061] The hearing device 4 and the hearing app 20 exchange data over a wireless link 24, e.g. a Bluetooth standard based wireless link 24. To this end, the hearing app 20 accesses a wireless transceiver (not shown), in particular a Bluetooth transceiver, of the mobile phone 22 in order to transmit data to and receive data from the hearing device 4.

[0062] The hearing app 20 comprises functions for remotely controlling, configuring and updating the hearing device 4. For this and other purposes, the hearing app 20 connects to a remote cloud service 26, e.g. using a cellular connection 28 and the Internet of the mobile phone 22.

[0063] Figure 2The internal structure of the first embodiment of the hearing system 2 is shown in more detail. As can be seen from this figure, the signal processor 14 comprises a pre-processing unit 30 which is designed for pre-processing the input audio signal I, in particular by beamforming and / or active noise cancellation, for denoising the input audio signal I. The signal processor 14 further comprises a main processing unit 32 which is, for example, designed for applying dynamic compression and frequency-dependent amplification to the (pre-processed) input audio signal I' output by the pre-processing unit 30.

[0064] Furthermore, the signal processor 14 comprises a signal analysis unit 34 which is designed for analyzing the pre-processed input audio signal I' (or the original input audio signal I). In particular, the analysis conducted by the signal analysis unit 34 comprises:

[0065] • voice activity detection (VAD), i.e. the detection of (any) speech in the input audio signal I' independent of a specific speaker,

[0066] • own voice analysis (OVD), i.e. the selective detection of the user's own speech in the input audio signal I', and

[0067] • scene classification, i.e. the analysis of the input audio signal I' in order to attribute the recorded sound to one of a plurality of pre-defined acoustic environments (sound classes), e.g. "speech in a quiet environment", "speech in a noisy environment", "music", etc.

[0068] Furthermore, the analysis unit 34 is designed for determining a plurality of pre-defined characteristic voice properties VP (e.g. pitch and harmonic structure) and / or speech properties SP (e.g. speech rate) from the input audio signal I' if the input audio signal I' contains speech.

[0069] The signal analysis unit 34 is designed for controlling the pre-processing unit 30 and the main processing unit 32 on the basis of the results of the signal analysis.

[0070] Preferably, the pre-processing unit 30, the main processing unit 32 and the signal analysis unit 34 are designed as software modules which are installed in and executable in the signal processor 14.

[0071] The hearing system 2 is designed for automatically performing a method for conducting speech understanding ability training, for which the hearing device 2, the hearing app 20 and, optionally, the cloud service 26 interact. The method is applicable to a user of the hearing system 2, so that the user is a participant 36 of the speech understanding ability training.

[0072] The method starts with a preparation phase in which speech and / or speech features of the communication partners of the participant 36 are collected. This preparation phase can be performed automatically during the operation of the hearing system 2. Alternatively, the preparation phase can be initiated by a user's command, which can be entered in the hearing app 20, for example.

[0073] In the preparation phase of the method, the microphone 8 of the hearing device 4 records the sound from the environment of the participant 36. The signal analysis unit 34 performs an analysis of the recorded sound. Here, the signal analysis unit 34 identifies whether speech of one or more communication partners of the participant 36 is present in the input audio signal I', for example by the fact that speech activity is detected without detecting the participant's 36 own speech. By way of example, two communication partners 38 and 40 are depicted in Figure 2

[0074] The signal analysis unit 34 extracts the speech of the communication partners 38, 40 from the input audio signal I' and determines speech features VP and / or speech features SP from the extracted speech. The signal analysis unit 34 sends these speech features VP and / or speech features SP to the hearing app 20, which stores this information in the first (data) memory 42. Optionally, the signal analysis unit 34 controls the de-noising of the input signal I by the pre-processing unit 30 in dependence on the analysis result of the scene classification, i.e. in dependence on the finding that the recorded sound corresponds to its sound class, to ensure sufficient quality of the extracted speech. For example, the de-noising is increased for the sound class "speech in a noisy environment" compared to the sound class "speech in a quiet environment".

[0075] The speaker recognition unit 44 (which is implemented here by way of example as part of the hearing app 20) analyzes the significant consistencies and differences of the speech features VP and / or speech features SP and thus identifies and distinguishes the contributions of different speakers (e.g. the communication partners 38 and 40). The speaker recognition unit 44 thereby determines the frequency and the time at which the respective speakers speak in order to thus assess the importance of the different speakers as communication partners of the participant 36. However, for the reasons described in detail above, the speaker recognition unit 44 selects the speaker who speaks most frequently or longest (e.g. the communication partner 40) as the second speaker of the method and the speaker who speaks second most frequently or longest (e.g. the communication partner 38) as the first speaker of the method.

[0076] The hearing app 20 sends the stored speech features VP and / or speech features SP of both the first speaker (communication partner 38) and the second speaker (communication partner 38) to the speech synthesis unit 46. In​Figure 2 In the example of Fig. 1, the speech synthesis unit 46 is implemented in the cloud service 26.

[0077] The speech synthesis unit 46 comprises a text-to-speech algorithm designed for creating a set of test audio sequences PS (phoneme stimuli), wherein each of the test audio sequences PS contains a synthesized speech of a combination of phonemes, which can include a consonant embedded in a vowel environment ("affa", "akka", "agga", etc.) or a vowel embedded in a consonant environment ("mom", "mam", "mem", etc.). Here, the speech synthesis unit 46 uses the voice properties VP and / or the speech properties SP provided by the hearing app 20 to synthesize the test audio sequences PS in a voice similar to one of the respective communication partners. In one embodiment of the method, the speech synthesis unit 46 can be adjusted so as to reproduce the voice and / or sound characteristics of the communication partners 38, 40 as closely as possible. In an alternative embodiment of the method, the speech synthesis unit 46 can be adjusted so as to synthesize the test audio sequences PS in a voice similar to, but not identical to, the voice of the respective speaker. Presenting the phoneme stimuli in a slightly modified voice compared to the known voice enables a very robust training benefit in real-world noisy environments, as the brain of the participant 36 benefits from training with multiple voices.

[0078] As a further input, the speech synthesis unit 46 receives from a database 47 a text representation T of the combination of phonemes for which the test audio sequences PS shall be created. In particular, the speech synthesis unit 46 creates a first set of test audio sequences PS whose voice and / or speech characteristics are similar to the voice and / or speech characteristics of the first speaker (communication partner 38) and a second set of test audio sequences PS whose voice and / or speech characteristics are similar to the voice and / or speech characteristics of the second speaker (communication partner 40). The speech synthesis unit 46 sends the first and second set of test audio sequences PS to the hearing app 20, which stores the test audio sequences PS in a second (data) memory 48.

[0079] The execution phase of the method is controlled by a training unit 50 implemented as part of the hearing app 20. In each of a number of subsequent training steps, the training unit 50 selects one of the test audio sequences PS (e.g. at random) from the data store 48 and feeds the selected test audio sequence PS to the main processing unit 32 of the hearing device 4 for processing and output to the participant 36 via the receiver 10. Here, the processing by the main processing unit 32 ensures that the selected test audio sequence PS is adapted to the individual requirements of the participant 36; for example, by processing to compensate for the hearing loss of the participant 36. The phoneme combination thus output to the participant 36 is referred to as the "presented phoneme stimulus" of the respective training step.

[0080] Simultaneously with the provision of the selected test audio sequence PS (i.e. the presented phoneme stimulus) to the participant 36, the training unit 50 outputs a multiple-choice request R on the display screen 52 of the mobile phone 22, which prompts the participant 36 to select the phoneme combination he / she thinks to have heard. For example, if the selected test audio sequence PS comprises the phoneme combination "agga", the multiple-choice request R can be presented in the following form:

[0081] "What did you hear?

[0082] o agga

[0083] o affa

[0084] o agga

[0085] o akka

[0086] Please select."

[0087] When the participant 36 has selected S one of the phoneme combinations specified in the request R (e.g. by clicking on the display screen 52, which can be implemented as a touch-sensitive display screen, for example), the training unit 50 outputs a feedback F to the participant 36 informing the participant 36 of the correctness of the selection S.

[0088] The feedback F can be positive feedback informing the participant 36 that his selection S was correct (if the participant 36 selected the phoneme combination understood to correspond to the presented phoneme stimulus) or negative feedback (if the participant 36 selected a phoneme combination understood to be different from the presented phoneme stimulus). The feedback F can be provided in a visually perceptible form via the display screen 52 of the mobile phone 22 or in an acoustically perceptible form via the main processing unit 32 and the receiver 10 of the hearing device 4, or preferably in a combination of these two forms.

[0089] In case of positive feedback, the training unit 50 outputs an audio message via the receiver 10, which can repeat the presented phoneme stimulus. For example, the audio sequence can contain the spoken sentence "You correctly selected the phoneme you heard: "agga"'. The training unit 50 displays a confirmation text message or symbol on the display screen 52.

[0090] In case of negative feedback, the audio message output by the training unit 50 via the receiver 10 can repeat the presented phoneme stimulus followed by the phoneme combination that the participant 36 indicated to have understood. For example, the audio message can contain the spoken sentence "Your selection was incorrect. You heard "agga", but you selected "akka"'. Here, the two phoneme combinations contained in the audio sequence are selected from a first set of audio sequences from which the training unit 50 selected the presented phoneme stimulus. Thus, the two phoneme combinations are synthesized so as to correspond to the voice of the same (first) speaker / communication partner 38. Likewise, the training unit 50 can display a corresponding text message or symbol on the display screen 52 as a further part of the negative feedback. However, the text message or symbol does not contain the presented phoneme stimulus to ensure that the participant 36 only hears, but does not see, the presented phoneme stimulus.

[0091] Then, the training unit 50 performs a further training step, which comprises the same actions as the training step described above. However, the training unit 50 bases the decision regarding the phoneme stimulus to be presented on the results of the previous training step.

[0092] If the selection S of the participant 36 was correct in the previous training step, the training unit 50 selects the phoneme stimulus to be presented from the first set of test audio sequences PS (corresponding to the voice of the first speaker / communication partner 38), but selects the phoneme combination independently of the phoneme stimulus presented in the previous training step.

[0093] However, if the selection S of the participant 36 was incorrect in the previous training step, the training unit 50 again selects the same phoneme combination (e.g. "agga"), but now is to take the corresponding test audio sequence (containing the phoneme combination) from the second set of test audio sequences PS (corresponding to the voice of the second speaker / communication partner 40). Thus, in the further training step, the same phoneme combination is presented again, but now spoken in a voice that is more familiar to the participant 36.

[0094] If the participant 36 again fails to select the correct phoneme combination, the training unit 50 can again present the phoneme combination at an increased volume and / or a reduced noise level in a further training step. Additionally or in a further step, the training unit 50 can display the phoneme combination as text on the display screen 52, so that the participant 36 can see and read the information he / she failed to understand.

[0095] Figure 3 A further variant of the hearing system 2 is shown, which corresponds to the embodiment of Figure 2 except for the differences mentioned below. Figure 3 The main difference of the embodiment of Figure 2 compared to the embodiment of Figure 3 is that the audio sequence PS is not created in a preparation phase of the method before the first training step is performed. Instead, the audio sequence PS is created in real-time in the performance step of the method. To this end, in the embodiment of the speech synthesis unit 46 is connected between the training unit 50 and the main processing unit 32 of the hearing device 4. It creates the test audio sequence PS based on the speech properties VP and / or the speech properties SP provided by the memory 42 and the respective text representation T of the phoneme combination to be presented provided by the training unit 50. The training unit 50 selects the respective text representation T from the database 47.

[0096] Figure 3 In the example of Figure 2 the speech synthesis unit 46 and the database 47 are implemented locally at the mobile phone 22 as part of the hearing app 20. However, in a variant of the embodiment of Figure 3 at least one of the speech synthesis unit 46 and the database 47 can be implemented in the cloud service 26 as in the example of Figure 2 .

[0097] Furthermore, in a further variant of the hearing system 2 at least one of the memory 42, the memory 48 and the training unit 50 can be implemented in the cloud service 26.

[0098] The speech synthesis unit 46 is preferably implemented as an artificial intelligence (AI) model; for example a conventional text-to-speech model such as WaveNet, Tacotron or a variational autoencoder (VAE) can be used.

[0099] In the example of Figure 4In a further embodiment of the hearing system 2 shown, the speech synthesis unit 46 comprises a deep neural network which is trained directly with the natural speech of the communication partner 38, 40 with whom the participant 36 communicates, in particular the natural speech of the communication partners 38 and 40. In this embodiment, the signal analysis unit 34 of the hearing device 4 does not determine the voice properties VP and / or the speech properties SP. Instead, it transmits a natural speech sequence NSS, i.e. a sequence of the input signal I' which is found to contain speech of one of the communication partners 38, 40, to the hearing app 20 which stores the natural speech sequence NSS in the memory 42. The memory 42 then provides the natural speech sequence NSS corresponding to one of the communication partners 38 and 40 to the speech synthesis unit 26 for training the neural network implemented therein in order to reproduce the voice and / or speech characteristics of the respective speaker in a later text-to-speech conversion. A plurality of profiles corresponding to the voice and / or speech characteristics of different speakers can be prepared and stored for the speech synthesis unit 26.

[0100] Any one of the embodiments of the hearing system 2 described above can be extended to contain and use test audio sequences PS corresponding to the voice and / or speech characteristics of more than two communication partners.

[0101] Those skilled in the art will appreciate that numerous changes and / or modifications can be made to the specific examples described above without departing from the spirit and scope of the present application as broadly described. The examples are, therefore, to be considered in all respects as illustrative and not restrictive.

[0102] List of reference signs

[0103] 2 hearing system

[0104] 4 hearing device

[0105] 6 housing

[0106] 8 microphone

[0107] 10 receiver

[0108] 12 battery

[0109] 14 signal processor

[0110] 16 sound channel

[0111] 18 tip

[0112] 20 hearing app

[0113] 22 mobile phone

[0114] 24 wireless link

[0115] 26 cloud service

[0116] 28 cellular connection

[0117] 30 pre-processing unit

[0118] 32 main processing unit

[0119] 34 signal analysis unit

[0120] 36 participant

[0121] 38 exchange partner

[0122] 40 exchange partner

[0123] 42 (data) storage

[0124] 44 speaker recognition unit

[0125] 46 speech synthesis unit

[0126] 47 database

[0127] 48 (data) storage

[0128] 50 training unit

[0129] 52 display screen

[0130] F feedback

[0131] I input audio signal

[0132] I' (pre-processed) input audio signal

[0133] NSS natural speech sequence

[0134] O output audio signal

[0135] PS test audio sequence

[0136] R multiple choice requirement

[0137] S selection

[0138] SP speech property

[0139] T textual representation

[0140] U supply voltage

[0141] VP voice property

Claims

1. A method for conducting speech comprehension ability training, - wherein, recording sounds from the environment of a participant (36); - wherein speech of a first speaker (38) different from the participant (36) is extracted from the recorded sounds; - wherein at least one characteristic voice property (VP) and / or speech property (SP) of the first speaker (38) is determined from the extracted speech; - wherein a plurality of first test audio sequences (PS) is created, wherein each of the plurality of first test audio sequences (PS) contains synthesized speech of a phoneme or a combination of phonemes and the speech is synthesized to comply with the at least one characteristic voice property (VP) and / or speech property (SP) of the first speaker (38); - wherein a first training step is conducted, in which, - one of the plurality of first test audio sequences (PS) is selected, converted to sound and output to the participant (36); - a response (S) of the participant (36) indicating a phoneme or a combination of phonemes understood by the participant (36) is collected; and - feedback (F) to the participant (36) is output to the participant (36) whether the phoneme or the combination of phonemes indicated as understood by the participant (36) corresponds to the phoneme or the combination of phonemes output to the participant (36) in the first training step.

2. The method according to claim 1, wherein, if in the first training step the phoneme or the combination of phonemes indicated as understood by the participant (36) does not correspond to the phoneme or the combination of phonemes output to the participant (36), the feedback (F) contains the speech sound of the phoneme or the combination of phonemes indicated as understood by the participant (36) and a repetition of the speech sound of the phoneme or the combination of phonemes output to the participant (36) in the first training step, wherein the speech sound of the phoneme or the combination of phonemes indicated as understood by the participant (36) is synthesized to comply with the at least one characteristic voice property (VP) and / or speech property (SP) of the first speaker (38).

3. The method according to claim 1, - wherein, extracting speech of a second speaker (40) different from the participant (36) from the recorded sounds; - wherein at least one characteristic voice property (VP) and / or speech property (SP) of the second speaker (40) is determined from the extracted speech; - wherein a plurality of second test audio sequences (PS) is created, wherein each of the plurality of second test audio sequences (PS) contains synthesized speech of a phoneme or a combination of phonemes and the speech is synthesized to comply with the at least one characteristic voice property (VP) and / or speech property (SP) of the second speaker (40); - wherein, if in the first training step the phoneme or the combination of phonemes indicated as understood by the participant (36) does not correspond to the phoneme or the combination of phonemes output to the participant (36), in a second training step, - selecting a test audio sequence (PS) of the plurality of second test audio sequences, converting it into sound and outputting it to the participant (36), wherein the selected test audio sequence (PS) of the plurality of second test audio sequences contains the same phoneme or phoneme combination as the test audio sequence (PS) output in the first training step; - collecting a response (S) of the participant (36) indicating the phoneme or phoneme combination understood by the participant (36); and - outputting feedback (F) to the participant (36) whether the phoneme or phoneme combination indicated as understood by the participant (36) corresponds to the phoneme or phoneme combination output to the participant (36) in the second training step.

4. The method according to claim 1, - wherein, extracting speech of a plurality of speakers (38, 40) different from the participant (36) from the recorded sound; - wherein the extracted speech is evaluated with respect to a frequency and / or a time period of speaking of each of the speakers (38, 40), and - wherein one of the plurality of speakers who speaks most frequently or longest is selected as the first speaker (38).

5. The method according to claim 3, - wherein, extracting speech of a plurality of speakers (38, 40) different from the participant (36) from the recorded sound; - wherein the extracted speech is evaluated with respect to a frequency and / or a time period of speaking of each of the speakers (38, 40), and - wherein one of the plurality of speakers who speaks most frequently or longest is selected as the first speaker (38), and - wherein the speaker who speaks most frequently or longest is selected as the second speaker (40), and the speaker who speaks second most frequently or second longest is selected as the first speaker (38).

6. The method according to claim 1, wherein denoising the recorded sound from the environment of the participant (36) before determining the at least one characteristic voice property (VP) and / or speech property (SP) of the respective speaker (38, 40) from the extracted speech.

7. The method according to claim 1, wherein, creating test audio sequences (PS) using artificial intelligence, in particular artificial neural networks.

8. The method according to claim 1, wherein recording sound from the environment of the participant (36) using a hearing device (4) worn on or in the ear of the participant (36), and outputting test audio sequences (PS) or each test audio sequence (PS) to the participant (36).

9. The method according to claim 8, wherein extracting speech of a speaker (38, 40) or each speaker (38, 40) different from the participant (36) from the recorded sound using the hearing device (4).

10. A method for conducting speech understanding ability training, - wherein, prompting at least one speaker (38, 40) different from the participant (36) to pronounce a plurality of predefined phonemes or phoneme combinations, - wherein the phonemes or phoneme combinations pronounced by the at least one speaker (38, 40) are recorded and stored as a plurality of test audio sequences (PS), wherein each of the plurality of test audio sequences (PS) contains a respective phoneme or phoneme combination, - wherein a training step is performed, in which - one of the plurality of test audio sequences (PS) is selected, converted into sound and output to the participant (36); - a response (S) of the participant (36) is collected, which indicates the phoneme or phoneme combination understood by the participant (36); and - feedback (F) is output to the participant (36), which indicates whether the phoneme or phoneme combination indicated as understood by the participant (36) corresponds to the phoneme or phoneme combination output to the participant (36) in the training step.

11. An apparatus, in particular a hearing system (2), for conducting speech understanding ability training, which is designed to automatically perform a method for conducting speech understanding ability training, the method comprising: - recording sound from an environment of a participant (36); - extracting speech of a first speaker (38) different from the participant (36) from the recorded sound; - determining at least one characteristic voice property (VP) and / or speech property (SP) of the first speaker (38) from the extracted speech; - creating a plurality of first test audio sequences (PS), wherein each of the plurality of first test audio sequences (PS) contains a synthesized speech of a phoneme or phoneme combination, and the speech is synthesized to comply with the at least one characteristic voice property (VP) and / or speech property (SP) of the first speaker (38); - performing a first training step, in which - one of the plurality of first test audio sequences (PS) is selected, converted into sound and output to the participant (36); - a response (S) of the participant (36) is collected, which indicates the phoneme or phoneme combination understood by the participant (36); and - feedback (F) is output to the participant (36), which indicates whether the phoneme or phoneme combination indicated as understood by the participant (36) corresponds to the phoneme or phoneme combination output to the participant (36) in the first training step.

12. The apparatus according to claim 11, wherein, If, in the first training step, the phoneme or the combination of phonemes indicated as understood by the participant (36) does not correspond to the phoneme or the combination of phonemes output to the participant (36), the feedback (F) comprises the speech sound of the phoneme or the combination of phonemes indicated as understood by the participant (36) and a repetition of the speech sound of the phoneme or the combination of phonemes output to the participant (36) in the first training step, wherein the speech sound of the phoneme or the combination of phonemes indicated as understood by the participant (36) is synthesized to comply with the at least one characteristic voice property and / or speech property of the first speaker (38).

13. The apparatus of claim 11, - wherein, extracting speech of a second speaker (40) different from the participant (36) from the recorded sound; - wherein at least one characteristic voice property (VP) and / or speech property (SP) of the second speaker (40) is determined from the extracted speech; - wherein a plurality of second test audio sequences (PS) is created, wherein each of the plurality of second test audio sequences (PS) comprises a synthesized speech of a phoneme or a combination of phonemes and the speech is synthesized to comply with the at least one characteristic voice property (VP) and / or speech property (SP) of the second speaker (40); - wherein, if in the first training step, the phoneme or the combination of phonemes indicated as understood by the participant (36) does not correspond to the phoneme or the combination of phonemes output to the participant (36), in a second training step, - selecting a test audio sequence (PS) of the plurality of second test audio sequences, converting it into sound and outputting it to the participant (36), wherein the selected test audio sequence (PS) of the plurality of second test audio sequences comprises the same phoneme or the same combination of phonemes as the test audio sequence (PS) output in the first training step; - collecting a response (S) of the participant (36) indicating a phoneme or a combination of phonemes understood by the participant (36); and - outputting feedback (F) to the participant (36) whether the phoneme or the combination of phonemes indicated as understood by the participant (36) corresponds to the phoneme or the combination of phonemes output to the participant (36) in the second training step.

14. The apparatus of claim 11, - wherein, extracting speech of a plurality of speakers (38, 40) different from the participant (36) from the recorded sound; - wherein the extracted speech is evaluated with respect to a frequency and / or a time period of speaking of each of the speakers (38, 40), and - wherein one of the plurality of speakers who speaks most frequently or longest is selected as the first speaker (38).

15. The apparatus of claim 13, - wherein, extracting speech of a plurality of speakers (38, 40) different from the participant (36) from the recorded sound; - wherein the extracted speech is evaluated with respect to the frequency and / or the time period at which each of the speakers (38, 40) spoke, and - wherein one of the speakers who spoke most frequently or longest is selected as the first speaker (38), and - wherein the speaker who spoke most frequently or longest is selected as the second speaker (40), and the speaker who spoke second most frequently or longest is selected as the first speaker (38).

16. The apparatus of claim 11, wherein, Before determining the at least one characteristic voice property (VP) and / or speech property (SP) of the respective speaker (38, 40) from the extracted speech, the recorded sound from the environment of the participant (36) is de-noised.

17. The apparatus of claim 11, wherein The test audio sequences (PS) are created using artificial intelligence, in particular artificial neural networks.

18. The apparatus of claim 11, wherein, The sound from the environment of the participant (36) is recorded using a hearing device (4) worn on or in the ear of the participant (36), and the test audio sequence (PS) or each test audio sequence (PS) is output to the participant (36).

19. The apparatus of claim 18, wherein, The speech of the speaker (38, 40) or each speaker (38, 40) different from the participant (36) is extracted from the recorded sound using the hearing device (4).

20. An apparatus, in particular a hearing system (2), for conducting speech understanding ability training, which is designed to automatically perform a method for conducting speech understanding ability training, the method comprising: - prompting at least one speaker (38, 40) different from a participant (36) to speak a plurality of predefined phonemes or phoneme combinations, - the phonemes or phoneme combinations spoken by the at least one speaker (38, 40) being recorded and stored as a plurality of test audio sequences (PS), wherein each of the plurality of test audio sequences (PS) contains a respective phoneme or phoneme combination, - conducting a training step in which - one of the plurality of test audio sequences (PS) is selected, converted into sound and output to the participant (36); - responses (S) of the participant (36) indicating the phonemes or phoneme combinations understood by the participant (36) are collected; and - feedback (F) is output to the participant (36) as to whether the phonemes or phoneme combinations indicated as understood by the participant (36) correspond to the phonemes or phoneme combinations output to the participant (36) in the training step.

Images