Methods, systems, media for wireless personal communications via a hearing device

Abstract

A method for wireless personal communication using a hearing system (10) with a hearing device (12) comprising monitoring and analyzing the acoustic environment of a user by the hearing device (12) to identify one or more talkers based on content-independent talker voiceprints saved in the hearing system (10); and presenting a user interface to the user for informing the user of the identified talkers and for establishing, joining or leaving a wireless personal communication connection between the hearing device (12) and one or more communication devices used by the one or more identified talkers.

Description

Technical Field

[0001] This invention relates to a method, computer program, and computer-readable medium for wireless personal communication using a hearing device worn by a user and equipped with at least one microphone and a sound output device. Furthermore, this invention relates to a hearing system comprising at least one hearing device of this type and optionally including a connected user device, such as a smartphone. Background Technology

[0002] Hearing devices are typically small and complex. They may include a processor, microphone, integrated speaker as a sound output device, memory, housing, and other electronic and mechanical components. Some examples of hearing devices are behind-the-ear (BTE), in-the-canal (RIC), in-the-ear (ITE), completely in-the-canal (CIC), and in-the-canal (IIC) devices. Based on hearing loss, aesthetic preferences, lifestyle needs, and budget, users may prefer one of these hearing devices over another.

[0003] Hearing devices from different users can be adapted to form a wireless personal communication network, which can improve communication (such as conversation or listening to someone’s speech) via voice with other hearing device users or people using any type of suitable communication device (such as a wireless microphone) in noisy environments.

[0004] The hearing device is then used as a head-mounted device, which receives the user's speech through its integrated microphone and makes the speech of other communication participants audible through its integrated speaker. For example, the speech audio stream is then transmitted from one user's hearing device to another user's hearing device, or generally in both directions. In this case, it is also known to use beamformers provided in the hearing device to improve the signal-to-noise ratio (SNR) in certain situations: if the speaker is in front of the user and if the speaker is not too far away (typically, closer than about 1.5 m).

[0005] In the prior art, several methods for automatically establishing wireless audio communication between hearing devices or other types of communication devices are known. A considerable amount of existing technology exists regarding automatic connection establishment based on the correlation between acoustic signals and digital audio streams. However, this approach is impractical for the hearing device network described herein, as the digital audio signals used for personal communication are not intended to be streamed prior to network connection establishment, and doing so would consume excessive power. Further methods either refer to connections triggered by linguistic content such as voice commands, or are based on analysis of the current acoustic environment or signals from sensors unrelated to speaker speech analysis. Summary of the Invention

[0006] One object of the present invention is to provide a method and system for wireless personal communication using a hearing device worn by a user and equipped with at least one microphone and a sound output device, which can further improve user comfort, signal quality and / or save energy compared to methods and systems known in the art.

[0007] These objectives are achieved through the subject matter of the independent claims. Further exemplary embodiments will be apparent from the dependent claims and the following description.

[0008] The first aspect of the invention relates to a method for wireless personal communication using a hearing device worn by a user and equipped with at least one integrated microphone and sound output device (e.g., a speaker).

[0009] This method can be computer-implemented and can be automatically executed by a hearing system, with the user's hearing device being part of the system. For example, the hearing system may include one or two hearing devices used by the same user. One or both hearing devices may be worn on and / or in the user's ear. The hearing device may be a hearing aid, suitable for compensating for the user's hearing loss. A cochlear implant may also be a hearing device. Optionally, the hearing system may further include at least one connected user device, such as a smartphone, smartwatch, or other device carried by the user and / or a personal computer.

[0010] According to one embodiment of the invention, the method includes monitoring and analyzing a user's acoustic environment via a hearing device to identify one or more speakers based on content-independent speaker voiceprints stored in the hearing system. The user's acoustic environment can be monitored by receiving audio signals from at least one microphone (e.g., at least one integrated microphone). The user's acoustic environment can be analyzed by evaluating the audio signals to identify one or more speakers (referred to herein as "speaker identification") based on content-independent speaker voiceprints stored in the hearing system.

[0011] According to one embodiment of the invention, the speaker identification is used as a triggering factor for a wireless personal communication connection that may automatically establish, join, or leave the user's hearing device and a corresponding communication device used by one or more speakers (also referred to herein as "other conversational participants") and capable of wirelessly communicating with the user's hearing device. In this document, the term "conversation" is intended to include any type of personal voice communication (i.e., not only a conversation between two people, but also talking in a group or listening to someone speak, etc.).

[0012] In other words, the basic idea of ​​the proposed method is to establish, join, or leave a network of hearing devices based on speaker recognition technology (i.e., speaker verification based on text or content-independent methods), or at least to inform the user of the possibility of such a connection. To this end, for example, hearing devices capable of wireless audio communication can expose the user's own content-independent voiceprint (e.g., the user's appropriate speaker model) so that another pair of hearing devices belonging to another user can compare it with the current acoustic environment.

[0013] Speaker recognition can be performed by identifying the characteristic frequencies, prosody, and / or dynamics of a speaker's speech. Speaker recognition can also be based on classification methods such as GMM, SVM, k-NN, Parzen window, and other machine learning and / or deep learning classification methods, such as DNN.

[0014] For example, the automatic activation of speaker-identification-based wireless personal communication connections described herein may be more suitable than manual activation by hearing device users, because manual activation may have the following disadvantages: First, users may find it difficult to know when activating such a wireless personal communication connection could be beneficial. Users may also forget about using the option. Secondly, it can be cumbersome for users to repeatedly activate the connection under the same circumstances. In this case, it would be easier to activate it automatically based on the situation. • Third, it can be very disturbing when a user forgets to deactivate the connection when they want to protect their privacy and are unaware that others are hearing their voice.

[0015] On the other hand, compared to known methods of automatic wireless connection activation further outlined above, the solution described herein can, for example, leverage the advantages that the speaker's hearing device possesses prior knowledge of the speaker's speech and is able to convey his voice signature (a content-independent speaker voiceprint) to the device of a potential conversation partner. This reduces complexity compared to methods known in the art and the number of inputs required. Essentially, the speaker recognition method described herein requires only acoustic and radio interfaces.

[0016] According to one embodiment of the present invention, a communication device capable of wirelessly communicating with a user's hearing device includes the hearing devices and / or wireless microphones of other people, i.e., the hearing devices and / or wireless microphones used by other participants in the conversation.

[0017] According to one embodiment of the invention, a beamformer is employed in the user's hearing device and / or the communication device of other conversation participants. This beamformer is specifically configured and / or tuned to improve the signal-to-noise ratio (SNR) of wireless personal communication between people who are not face-to-face (i.e., the speaker is not in front of the user) and / or who are separated by more than 1 m, more than 1.5 m, or more than 2 m. Therefore, SNR can be significantly improved under adverse listening conditions compared to solutions known in the art, where beamformers typically only improve SNR in certain cases where the speaker is in front of the user and not too far away (approximately less than 1.5 m).

[0018] According to one embodiment of the invention, a user's own content-independent voiceprint can also be stored in the hearing system and shared (i.e., exposed and / or transmitted) wirelessly with communication devices used by potential conversation participants, so that the user can be identified based on their own content-independent voiceprint. The voiceprint may also be stored externally to the device, for example, on a server or cloud-based service. For example, a user's own content-independent voiceprint can be stored in the non-volatile memory (NVM) of a user device (e.g., a smartphone) connected to the user's hearing device or hearing system for permanent availability. Content-independent speaker voiceprints of potential other conversation participants can also be stored in non-volatile memory, for example, if there are other important individuals, such as close relatives or colleagues. However, storing the content-independent speaker voiceprints of potential conversation participants in volatile memory for availability only when needed may also be suitable, for example, in use cases such as meetings or other public events.

[0019] According to one embodiment of the present invention, a user's own content-independent voiceprint can be shared with the communication devices of potential conversation participants through one or more of the following methods:

[0020] When a user's hearing device pairs with another participant's communication device for wireless personal communication, it can be shared through the exchange of the user's own content-independent voiceprint and the corresponding content-independent speaker's voiceprint. Here, pairing between different users' hearing devices can be done manually or automatically, for example, using Bluetooth, and means merely preparing for wireless personal communication, not activating it. In other words, the connection is not necessarily automatically activated by a single paired hearing device. During pairing, a speech model stored in one hearing device can be loaded into the other, and a connection can be established when the speech model is recognized and optionally meets other conditions (e.g., poor SNR) as described below.

[0021] Additionally or alternatively, a user's own content-independent voiceprint may also be shared by: periodic broadcasts performed by the user's hearing device at predetermined time intervals, and / or by sending it upon request from the communication devices of potential other conversation participants.

[0022] According to one embodiment of the invention, the user's own content-independent voiceprint is obtained using specialized speech feature extraction and voiceprint modeling equipment, such as during a fitting session in the office of a hearing healthcare professional or in another medical or industrial office or institution. This can have the advantage that the computational complexity of the model can be pushed onto specialized equipment in that office or institution, such as a fitting station. It can also have the advantage or disadvantage that the model / voiceprint is created in a quiet environment.

[0023] Additionally or alternatively, a user's own content-independent voiceprint can also be obtained by extracting speech features during real-world use cases (also known as OVPU) where the user is speaking (e.g., making a phone call). Specifically, a beamformer provided in the hearing device can be tuned to pick up the user's own speech and filter out ambient noise during such real-world use cases. A potential advantage of this approach is that the voiceprint / model can improve over time in real-world scenarios. The speech model (voiceprint) can also be calculated online: via the hearing device itself or the user's phone or other connected devices.

[0024] If model computation is switched to a mobile phone or other connected user device, at least two different approaches can be considered. For example, in a real-world use case where a user is speaking and voiceprint modeling is being performed using a connected user device, the user's own content-independent voiceprint can be acquired using the user's hearing device and / or the connected user device for speech feature extraction. The user's hearing device may then extract the speech features and send them to the connected user device, which then computes or updates the voiceprint model and optionally sends it back to the hearing device. Alternatively, the connected user device can use a mobile application (e.g., a phone app) that, for example, monitors the user's calls and / or other speaking activities with the user's consent and performs the speech feature extraction portion attached to the voiceprint modeling.

[0025] According to embodiments of the invention, in addition to speaker identification described above and below, one or more additional conditions related to the wireless personal communication are monitored and / or analyzed in the hearing system. In this embodiment, the steps of automatically establishing, joining, and / or leaving wireless personal communication connections between the user's hearing device and the corresponding communication devices of other conversation participants also depend on these additional conditions, which are not based on speech recognition. For example, these additional conditions may relate to acoustic quality, such as the signal-to-noise ratio (SNR) of the microphone signal, and / or any other factors or criteria related to deciding to initiate or terminate the wireless personal communication connection.

[0026] For example, these additional conditions could include the ambient signal-to-noise ratio (SNR) so that wireless communication is automatically switched whenever the ambient SNR of the microphone signal is too poor for conversation, and vice versa. As conditions, additional conditions could also include the existence of a predefined environmental scenario relating to the user and / or other people and / or surrounding objects and / or weather (e.g., the user and / or other people in a car or outdoors, wind noise, etc.). For example, such a scenario could be automatically identified by a corresponding classifier (sensors and / or software) provided by the hearing device or hearing system.

[0027] According to one embodiment of the invention, once a user's hearing device establishes a wireless personal communication connection with another speaker's communication device, the user's hearing device continuously monitors and analyzes the user's acoustic environment and terminates the wireless personal communication connection if the speaker's voiceprint is not further identified for a period of time (e.g., a predetermined time period, such as one minute or several minutes). Thus, for example, after the user or other conversation participants have left the conversation room, the user's privacy can be protected from further overhearing by other conversation participants. Furthermore, the automatic interruption of the wireless audio stream when the speaker's voice is no longer recognized also helps save energy for the hearing device or system.

[0028] According to one embodiment of the invention, if a user's hearing device establishes wireless personal communication connections with the communication devices of multiple other dialogue participants, the user's hearing device continues to monitor and analyze the user's acoustic environment and, depending on at least one predetermined sorting criterion, interrupts the wireless personal communication connections to some of these communication devices to form smaller dialogue groups. The aforementioned number can be a predetermined large number of dialogue participants, such as 5, 7, 10, or more. For example, it can be preset in the hearing system or device and / or individually selectable by the user. At least one predetermined sorting criterion can, for example, include one or more of the following: dialogue (i.e., content-related) overlap; directional gain determined by the user's hearing device to characterize the orientation of the user's head relative to the respective other dialogue participants; and spatial distance between the user and the respective other dialogue participants.

[0029] According to one embodiment of the invention, the method includes presenting a user interface to a user for notifying the user of an identified speaker and for establishing, joining, or leaving a wireless personal communication connection between the hearing device and one or more communication devices used by one or more identified speakers. The user interface may be presented as an acoustic user interface by the hearing device itself and / or as a graphical user interface by another user device (e.g., a smartphone).

[0030] A further aspect of the invention relates to a computer program for wireless personal communication using a hearing device worn by a user and equipped with at least one microphone and a sound output device, the program being adapted, when executed by a processor, to perform the steps of the methods described above and below, and to a computer-readable medium for storing such a computer program.

[0031] For example, a computer program can execute in the processor of a hearing device, which may be worn behind the ear by a person. A computer-readable medium may be the memory of the hearing device. The computer program can also execute in the processor of a connected user device, such as a smartphone or any other type of mobile device, which may be part of the hearing system, and the computer-readable medium may be the memory of the connected user device. Steps of the method can also be executed by the hearing device, and other steps of the method can be executed by the connected user device.

[0032] Generally, computer-readable media can be floppy disks, hard disks, USB (Universal Serial Bus) storage devices, RAM (Random Access Memory), ROM (Read-Only Memory), EPROM (Erasable Programmable Read-Only Memory), or flash memory. Computer-readable media can also be data communication networks that allow the download of program code, such as the Internet. Computer-readable media can be non-transitory or transient.

[0033] Another aspect of the invention relates to a hearing system comprising a hearing device worn by a user of a hearing device, as described above and below, wherein the hearing system is adapted to perform the methods described above and below. As an example, the hearing system may also include a second hearing device worn by the same user and / or a connected user device used by the same user, such as a smartphone or other mobile device or personal computer.

[0034] According to one embodiment of the present invention, a hearing device includes: a microphone; a processor for processing signals from the microphone; a sound output device for outputting the processed signals to the ear of a user of the hearing device; and a transceiver for exchanging data with a communication device used by other participants in a conversation and optionally with a connected user device and / or with another hearing device worn by the same user.

[0035] It must be understood that the features described above and below can be features of the computer programs, computer-readable media, and hearing systems described above and below, and vice versa.

[0036] These and other aspects of the invention will become apparent and will be elucidated with reference to the embodiments described below. Attached Figure Description

[0037] Hereinafter, embodiments of the present invention will be described in more detail with reference to the accompanying drawings.

[0038] Figure 1 A hearing system according to an embodiment of the present invention is illustrated schematically.

[0039] Figure 2 This illustration shows an example of two conversational participants (Alice and Bob) talking to each other via a wireless connection provided by their hearing devices.

[0040] Figure 3 An embodiment of the invention is shown for use via Figure 1 A flowchart illustrating a method for wireless personal communication using hearing devices within a hearing system.

[0041] Figure 4 A schematic block diagram of the speaker recognition method is shown.

[0042] Figure 5 A schematic block diagram illustrating the creation of a user-defined content-independent voiceprint according to an embodiment of the present invention is shown.

[0043] Figure 6 A schematic block diagram illustrating the speaker verification according to an embodiment of the present invention and the automatic establishment or closure of a wireless communication connection to the speaker's communication device based on the result of speaker identification is shown.

[0044] The reference symbols used in the accompanying drawings and their meanings are listed in summary form in the reference symbol list. In principle, the same parts have the same reference numerals in the drawings. Detailed Implementation

[0045] Figure 1 A hearing system 10 is schematically illustrated, comprising a hearing device 12 in the form of an behind-the-ear device worn by a hearing device user (not shown), and a connected user device 14, such as a smartphone or tablet. It should be noted that the hearing device 12 is a particular embodiment and the methods described herein can also be performed by other types of hearing devices, such as in-the-ear devices.

[0046] The hearing device 12 includes a postauricular component 15 and a component 16 to be placed in the user's ear canal. Components 15 and 16 are connected by a tube 18. Component 15 includes a microphone 20, a sound processor 22, and a sound output device 24, such as a speaker. The microphone 20 can acquire ambient sounds and generate sound signals, the sound processor 22 can amplify the sound signals, and the sound output device 24 can generate sound that is guided into the user's ear canal through the tube 18 and the in-ear component 16.

[0047] The hearing device 12 may include a processor 26 adapted to adjust parameters of a sound processor 22 such that the output volume of the sound signal is adjusted based on the input volume. These parameters can be determined by a computer program running in the processor 26. For example, using a knob 28 of the hearing device 12, a user can select a modifier (e.g., bass, treble, noise suppression, dynamic volume, etc.) and select the level and / or value of these modifiers, based on which adjustment commands can be created and processed as described above and below. In particular, processing parameters can be determined based on the adjustment commands, and based on this, for example, the frequency-dependent gain and dynamic volume of the sound processor 22 can be changed. All these functions can be implemented as computer programs stored in the memory 30 of the hearing device 12, which can be executed by the processor 22.

[0048] The hearing device 12 also includes a transceiver 32 adapted to wirelessly communicate with a transceiver 34 of a connected user device 14, which may be a smartphone or tablet computer. It is also possible to adjust the aforementioned regulators and their levels and / or values ​​using the connected user device 14, and / or to generate adjustment commands using the connected user device 14. This can be performed by a computer program running in the processor 36 of the connected user device 14 and stored in the memory 38 of the connected user device 14. The computer program can provide a graphical user interface 40 on the display 42 of the connected user device 14.

[0049] For example, to adjust a regulator such as volume, the graphical user interface 40 may include a control element 44 such as a slider. When the user adjusts the slider, an adjustment command can be generated, which will change the sound processing of the hearing device 12, as described above and below. Alternatively or additionally, the user can adjust the regulator using the hearing device 12 itself, for example, via a knob 28.

[0050] User interface 40 may also include indicator element 46, which, for example, displays the current listening status.

[0051] In addition, the transceiver 32 of the hearing device 12 is adapted to allow wireless personal communication between the hearing device 12 of the user and the hearing devices of other people via voice, so as to improve / enable their conversations in adverse acoustic conditions (such as noisy environments) (this includes not only conversations between two people, but also group conversations or listening to someone’s speech, etc.).

[0052] This is Figure 2 The image schematically depicts an example of two conversational participants (Alice and Bob) conversing with each other via a wireless connection provided by their hearing devices 12 or corresponding 120. Figure 2 As shown, hearing devices 12 and 120 are used as headsets; they use their integrated microphones to pick up the user's speech and make the speech of other participants in the communication audible through integrated speakers. Figure 2 As shown by the dashed arrows, the voice audio stream is then wirelessly transmitted from one user's (Alice's) hearing device 12 to another user's (Bob's) hearing device 120, or generally in both directions.

[0053] Figure 1 The hearing system 10 shown is adapted to perform methods for wireless personal communication (e.g., as...) using a hearing device 12 worn by a user and equipped with at least one integrated microphone 20 and a sound output device 24 (e.g., a speaker). Figure 2 (As shown).

[0054] Figure 3 An example flowchart of the method is shown. The method can be in... Figure 1 The computer-implemented method that is automatically executed in the hearing system 10.

[0055] In the first step S100 of the method, the user’s acoustic environment is monitored and analyzed by at least one microphone 20 in order to identify one or more speakers based on the content-independent speaker voiceprints of one or more speakers stored in the hearing system 10 (“speaker identification”).

[0056] In the second step S200 of the method, the speaker identification is used as a triggering factor to automatically establish, join or leave the user's hearing device 12 and a corresponding communication device (e.g., hearing device or wireless microphone) used by one or more speakers (also referred to as "other dialogue participants") and capable of wirelessly communicating with the user's hearing device 12.

[0057] In step S200, a user interface may also be presented to the user first, which notifies the user of the identified speaker and establishes the connection. Using the user interface, the hearing device may also be triggered by the user to join or leave a wireless personal communication connection between the hearing device (12) and one or more communication devices used by one or more identified speakers.

[0058] In an optional third step S300 of the method (which may also be performed before the first step S100 and the second step S200), the user’s own content-independent voiceprint is obtained and stored in the hearing system 10.

[0059] In the optional fourth step S400, the user's own content-independent voiceprint stored in the hearing system 10 is shared (i.e. exposed and / or transmitted) wirelessly to the communication devices of potential other dialogue participants so that they can identify the user as the speaker based on the user's own content-independent voiceprint.

[0060] In the following text, reference will be made to Figures 4 to 6 A more detailed description also includes each of the possible sub-steps in steps S100-S400. For example, some or all of steps S100-S400 or their sub-steps may be performed simultaneously or repeated periodically.

[0061] First, the analysis of the user's monitored acoustic environment, performed by the hearing system 10 in step S100 and represented as speaker identification, will be explained in more detail:

[0062] Speaker recognition technology is known in other technological fields as well. For example, it is commonly used in biometric authentication applications and forensics, and is often used to identify suspects in recorded calls (see, for example, JH Hansen and T. Hasan, "Speaker Recognition by Machines and Humans: A tutorial review," in IEEE Signal Processing Magazine (Volume: 32, Issue: 6), 2015).

[0063] like Figure 4As illustrated in the diagram, the speaker recognition method may include two stages:

[0064] 1) Training phase S110, in which the speaker's speech is modeled (as an example of generating a speaker's voiceprint unrelated to the above content); and

[0065] 2) Testing phase S120, in which unknown language segments are tested against the model (in order to identify the speaker as described above).

[0066] The likelihood that the test segment was generated by the speaker is then calculated and can be used to make decisions about the speaker's identity.

[0067] Therefore, as Figure 4 As shown, the training phase S110 may include: a "feature extraction" sub-step S111, in which the speaker's speech features are extracted from his speech samples, and a "speaker modeling" sub-step S112, in which the extracted speech features are used for content-independent speaker voiceprint generation. The testing phase S120 may further include: a "feature extraction" sub-step S121, in which the speaker's speech features are extracted from his speech samples obtained based on the monitored user's acoustic environment, followed by a "scoring" sub-step S122, in which the aforementioned likelihood is calculated, and a "decision" sub-step S123, in which a decision is made on whether to identify the corresponding speaker based on the score / likelihood.

[0068] Regarding the aforementioned speech features, one of the most popular speech features used in speaker recognition is called the Mel-Frequency Cepstrum Coefficient (MFCC) because it efficiently separates speech content from speech. In Fourier analysis, the cepstrum is the result of the inverse Fourier transform of the logarithm of the signal's spectrum. The Mel frequency is very close to the Bark domain, which is commonly used in hearing devices. It involves grouping the acoustic frequency bins on a logarithmic scale to reduce the dimensionality of the signal. In contrast to the Bark domain, overlapping triangular filters are used to group the frequencies. If the hearing device has already implemented the Bark domain, the Bark-Frequency Cepstrum Coefficient (BFCC) can be used as a feature, which will save some computation. For example, F. uR SKAM & GS Chandar Kumar's "Analysis of MFCC and BFCC in a Speaker Identification System" published at iCoMET 2018 compared the performance of speaker identification based on MFCC and BFCC and revealed that speaker identification based on BFCC is also generally applicable.

[0069] The cepstral coefficients can then be calculated as follows:

[0070] Where X(f) is the (Mel or Barker) frequency domain representation of the signal, and It is the inverse Fourier transform. For example, further insights into the cepstrum are given in the following literature: RWS Alan V. Oppenheim, "From Frequency to Quefrency: A History of the Cepstrum," IEEE Signal Processing Magazine, no. Sept., pp. 95-106, 2004.

[0071] It should be noted here that sometimes the inverse Fourier transform is replaced by the discrete cosine transform (DCT), which may more aggressively reduce the dimensionality. In both cases, embedding suitable digital signal processing techniques with hardware support for computation is generally considered feasible.

[0072] Other speech features that may be included alternatively or additionally in steps S111 and S121 to improve recognition performance may be, for example, one or more of the following:

[0073] • LPC coefficients (Linear Predictive Coding Coefficients)

[0074] • Pitch

[0075] • timbre

[0076] exist Figure 4 In step S112, the extracted speech features are used to construct a model that best describes the observed speech features of a given speaker.

[0077] Several modeling techniques can be found in the literature. The most commonly used is the Gaussian Mixture Model (GMM). A GMM is a weighted sum of several Gaussian PDFs (probability density functions), each consisting of a mean vector, a weight vector, and a mean vector. Figure 4 The covariance matrix is ​​represented by the calculations performed during the training phase S110. If some of these computational steps are too time-consuming or energy-intensive, or if they are too expensive to implement in the hearing device 12, they can also be swapped to the user device 14 connected to the hearing system 10 (see [link to documentation]). Figure 1 And / or offline execution (i.e., not performed in real time during the conversation). In other words, as will be presented below, model computation may be done offline.

[0078] On the other hand, the calculation of the likelihood of matching an unknown test segment with a given loudspeaker model (see...) Figure 4 Step S122) may need to be performed in real time by the hearing device. For example, the calculation may need to be performed as in... Figure 3 The conversation between Alice and Bob is conducted by their hearing devices 12 or corresponding 120 or by their connected user devices 14 (e.g., smartphones) (see [link]). Figure 1 ).

[0079] In this example, the likelihood to be calculated is equivalent to the speech feature vector observed in a given speech model λ. x The probability (given the speech model λ is the content-independent speaker voiceprint stored in the hearing system 10). For the Gaussian mixture mentioned above, this means calculating the probability as follows:

[0080] The meanings of the variables are as follows:

[0081] g = 1...M Gaussian component index

[0082] π g No. g The weights of the Gaussian mixture

[0083] N-dimensional Gaussian function

[0084] µ g No. g The average vector of Gaussian mixtures

[0085] ∑ g No. g The covariance matrix of a Gaussian mixture

[0086] The size of K eigenvectors

[0087] Calculating the likelihood using approximately 10 reasonable numbers of features can be too time-consuming or expensive for hearing devices. Therefore, to effectively reduce this complexity, the following different methods can be further implemented in the hearing system 10:

[0088] One approach is to simplify the model to a multivariate Gaussian (M=1), where: o Features are independent, with different means but equal variances (∑=σ) 2 ·I) or o The characteristic covariance matrices are equal (∑ i = ∑, Ɐi)

[0089] In these cases, the discriminant function simplifies to a linear separator (hyperplane), and the feature locations need to be computed as the linear separator (see below for more details on this).

[0090] • A Support Vector Machine (SVM) classifier can be used for speaker identification in step S120. The idea here is to separate the speaker model from the background using a linear decision boundary; also known as a hyperplane. Additional complexity is then added during the training phase in step S110, but the testing in step S120 will be greatly simplified because the observed feature vectors can be tested against a linear function. See the description below for testing with a linear classifier.

[0091] • Depending on the overall performance, suitable nonparametric density estimation can also be achieved, for example, using k-NN and Parzen windows.

[0092] As described above, by using the linear classifier described above, the complexity of likelihood calculation in step S120 can be greatly reduced.

[0093] In other words, the output of a linear classifier is given by the following equation:

[0094] The meanings of the variables are as follows:

[0095] g Nonlinear activation function

[0096] x Observed speech feature vectors

[0097] w Predetermined weight vector

[0098] w 0 Predetermined scalar deviation.

[0099] If g in the above formula is a sign function, then Figure 4 The decision in step S123 is given by the following formula:

[0100] As is readily apparent, the decision complexity is very low in the case of linear classifiers. That is, it is on the order of K MACs (multiplication-accumulation), where K is the size of the speech feature vector.

[0101] refer to Figure 5 Used to create user-defined content-independent voiceprints (see Figure 3 The training phase of step S300 (see step S300 in the text) Figure 4The specific application and implementation of step S110 will be explained.

[0102] As mentioned above, a user's own voice signature (content-independent voiceprint) can be obtained under different circumstances, for example:

[0103] - During a fitting session at the office of a hearing healthcare professional.

[0104] Therefore, the computational complexity of the model can be pushed to the fitting table. However, the model was created in a quiet environment.

[0105] - During its own voice pickup (OVPU) use cases (such as a call), the beamformer of the hearing device can be tuned to pick up the user's own voice and filter out ambient noise.

[0106] Therefore, the model can be improved over time in real-life situations. However, the model typically requires online computation, i.e., while the user is using his hearing device 12. This can be achieved either within the hearing device 12 itself or via the user's telephone (as...). Figure 1 (Example of user connection device 14) is executed.

[0107] It should be noted that if the model calculations are pushed to the mobile device, Figure 1 The hearing system 10 can achieve at least two methods:

[0108] 1) The hearing device 12 extracts features and sends them to the telephone. The telephone then calculates / updates the speaker model and sends it back to the hearing device 12.

[0109] 2) The phone app listens to calls with the user's consent and processes the feature extraction part that is added to the modeling.

[0110] Figure 5 The sub-steps of step S300 are schematically illustrated. In sub-step S301, the ambient acoustic signals acquired by the microphones M1 and M2 of the user's hearing device 12 while the user is speaking are preprocessed in any suitable manner. For example, the preprocessing may include noise cancellation (NC) and / or beamforming (BF), etc.

[0111] Optionally, detection of the user's own voice activity can be performed in sub-step S302 to ensure that the user is speaking, for example by recognizing a call connection with another person and / or by recognizing the direction of the acoustic signal from the user's mouth.

[0112] Similar to the reference above Figure 4The steps S111 and S112 are described in general. Then, in step S311, the user's speech features are extracted, and then in step S312, the user's speech is modeled, that is, his own content-independent voiceprint is created.

[0113] In step S314, the model of the user's speech can then be stored in non-volatile memory (NVM) (e.g., in the NVM of the hearing device 12 or the NVM of the connected user device 14) for future use. For use by the communication devices of other dialogue participants, it can be processed in step S400 (see...) Figure 3 This information can be shared with them, for example, via transceiver 32 of the user's hearing device 12. In step S400, the model can:

[0114] • Exchanging hearing devices between different people during pairing in a wireless personal communication network; and / or

[0115] • Broadcast regularly; and / or

[0116] • Whenever a hearing device is available to access an existing wireless personal communication network or to create a new wireless personal communication network, it is sent in Bluetooth Low Energy scan response mode as requested.

[0117] like Figure 5 As shown, sharing the user's own speech model with the devices of potential other dialogue participants in step S400 can also be implemented additionally, depending on whether the user is speaking, as detected in step S302. Therefore, energy can be saved by avoiding unnecessary model sharing, for example, when the user does not intend to speak, such as when he / she is simply listening to a speech or report by another speaker.

[0118] refer to Figure 6 The following examples will be used to explain and further illustrate the specific application during the testing phase (see [link]). Figure 4 Step S120) verifies the speaker through the user's hearing system 10; and, depending on the result of the speaker identification, automatically establishes or disconnects a wireless communication connection with the speaker's communication device (see step S120). Figure 3 Step S200 in the process.

[0119] In a face-to-face conversation between two people equipped with hearing devices capable of digital audio radio transmission, for example... Figure 2 In the case of Alice and Bob, the roles of "speaker" and "listener" can be defined at specific times during the dialogue. The listener is defined as the person who receives the speaker's voice. Figure 2At the specific moment shown, Alice is the “speaker” as the sound waves AW are shown, leaving her mouth and being received by the microphone 20 of her hearing device 12 in order to wirelessly transmit the content to Bob (who is the “listener” in this case).

[0120] Figure 6 The testing phase activities are performed by listening. It is based on signals received by microphones M1 and M2 of the user's hearing device 12 when they monitor the user's acoustic environment. In sub-step S101, the acoustic signals received by microphones M1 and M2 can be preprocessed in any suitable manner, such as noise cancellation (NC) and / or beamforming (BF). Figure 6 The listening process involves extracting speech features from the acoustic signal of interest, i.e., the beamformer signal output in this example, and calculating the likelihood using a known speaker model stored in the NVM. For example, speaker speech features can be extracted in step S121, and the likelihood can be calculated in step S122 to satisfy the decision regarding speaker identification in step 123, similar to the above reference. Figure 4 The steps described.

[0121] like Figure 6 As shown, the speaker recognition procedure may optionally include: an additional sub-step S102, “speaker speech activity detection”, wherein the presence of speaker speech may be detected before its features are extracted in step S121; and an additional sub-step S103, wherein a speaker speech model (content-independent voiceprint) (e.g., which is stored in non-volatile memory (NVM)) is provided to a decision unit, in which the analysis of steps S122 and S123 is performed.

[0122] As described above, in step S200 (see also...) Figure 2 In this process, the speaker identification performed in steps S122 and S123 is used as a trigger for automatically establishing, joining, or leaving a wireless personal communication connection between the user's hearing device 12 and the corresponding communication device of the identified speaker. This connection can be implemented by including a further sub-step S201, which can help further improve the wireless personal communication. For example, this might include monitoring additional conditions such as signal-to-noise ratio (SNR) or noise floor estimation (NFE).

[0123] The following sections will describe some examples of different use cases where the proposed method may be beneficial:

[0124] In step S200, a wireless personal communication stream is established:

[0125] If the listener's hearing system 10 detects that the identified speaker's device is known to be wireless network compatible, the listener's hearing device 12 or system 10 may request to establish a wireless network connection to the speaker's device or join an existing connection (if any), depending on acoustic parameters such as ambient signal-to-noise ratio (SNR) and / or the results of a classifier in the hearing device 12, which can identify scenarios such as people inside a vehicle, outdoors, and wind noise, so that decisions are made based on the identified scenario.

[0126] Leave the wireless personal communication network in step S200:

[0127] While consuming digital audio streams over the network, the listener's hearing device 12 continues to analyze the acoustic environment. If the active speaker's voice signature is not present in the acoustic environment for a certain period of time, the hearing device 12 can disconnect from the wireless network connection with that speaker's device to maintain privacy and / or save energy.

[0128] In step S200, the wireless personal communication group is split:

[0129] If a wireless personal communication network can grow automatically as users join, it may also split itself into smaller networks. If groups of four to six people can be identified in a suitable way, it is possible to break down and separate conversation participants into such smaller conversation groups within a hearing device network.

[0130] In this situation, a person will naturally turn their head towards the group they are interested in, which has an advantage in terms of directional gain. Therefore, when several people are speaking simultaneously in a group, the listener's hearing device may be able to rank the speakers according to their relative gain.

[0131] Based on this sequencing and dialogue overlap, the hearing device can decide to discard the stream from speakers who are further away.

[0132] In short, the novel method disclosed herein can be performed by a system that is a combination of a hearing device and a connected user device (e.g., a smartphone, personal or tablet computer). For example, a smartphone or computer can connect to a server that provides a speech model / speech imprint (referred to herein as a "content-independent voiceprint"). The analysis described herein (i.e., one or more analytical steps, such as speech feature extraction, speech model development, speaker identification, evaluation of further conditions (e.g., SNR)) can be performed in the hearing device and / or it can be performed in the connected user device. The speech model / imprint can be stored in the hearing device or the connected user device. The comparison between the detected speech model and the stored speech model can be implemented / performed in the hearing device and / or the connected user device.

[0133] While the invention has been detailed and described in the accompanying drawings and foregoing description, such illustrations and descriptions are intended to be illustrative or exemplary rather than restrictive; the invention is not limited to the disclosed embodiments. Other variations of the disclosed embodiments can be understood and implemented by those skilled in the art and by practice of the claimed invention, through study of the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other elements or steps, and the words "a" or "an" do not exclude multiple. A single processor or controller or other unit can perform the functions of several items recited in the claims. The fact that certain measures are recited in mutually different dependent claims does not mean that a combination of these measures cannot be used advantageously. Any reference numerals in the claims should not be construed as limiting the scope. List of reference numerals 10. Hearing System 12, 120 hearing devices 14 Connected user equipment 15. Behind-the-ear components 16. Intraacular components 18 tubes 20, M1, M2 microphones 22 sound processors 24. Audio output devices 26 processors 28 knobs 30 Memory 32 transceivers 34 Transceivers 36 processors 38. Memory 40 Graphical User Interface 42 monitors 44 Control elements, slider 46 Indicator Elements AW sound waves

Claims

1. A method for wireless personal communication using a hearing system, the hearing system comprising a hearing device worn by a user, the method comprising: The user's acoustic environment is monitored and analyzed by the hearing device to identify one or more other dialogue participants based on the content-independent speaker voiceprint stored in the hearing system. as well as Depending on speaker identification, a wireless personal communication connection is established, joined, or left between the hearing device and one or more communication devices used by one or more other identified conversation participants. The user's own content-independent voiceprint is also stored in the hearing system and shared wirelessly with communication devices used by potential dialogue participants, so that the communication devices can identify the user based on the user's own content-independent voiceprint. When the user's hearing device is paired with the communication device of another dialogue participant for wireless personal communication, the user's own content-independent voiceprint is shared in the following way: the exchange of the user's own content-independent voiceprint and the corresponding content-independent speaker's voiceprint.

2. The method of claim 1, further comprising: The communication device capable of wirelessly communicating with the user's hearing device includes the hearing device and / or a wireless microphone used by other participants in the conversation; and / or Beamformers, specifically configured and / or tuned to improve the signal-to-noise ratio of wireless personal communication between people who are not face-to-face standing and / or more than 1.5 m apart, are employed in the user's hearing device and / or in the communication devices of the other participants in the conversation.

3. The method as described in claim 1, wherein, The user's own content is unrelated to the voiceprint: Stored in the non-volatile memory of the user's hearing device or the non-volatile memory of a connected user device; and / or Share with potential conversation participants via one or more of the following communication devices: Regular broadcasts performed at predetermined time intervals by the user's hearing device; The user sends their own content-independent voiceprint upon request from the communication devices of potential other participants in the conversation.

4. The method of claim 1, wherein, The user's own content-irrelevant voiceprint is obtained through the following methods: During the fitting session, in the office of a hearing healthcare professional, a professional voice feature extraction and voiceprint modeling device is used. and / or During the actual use case of the user speaking, speech feature extraction is performed using the user's hearing device and / or connected user devices.

5. The method of claim 4, wherein, The user's own content-irrelevant voiceprint is obtained through the following methods: During the actual use case where the user is speaking, speech feature extraction is performed using the user's hearing device and / or the connected user device, and voiceprint modeling is performed using the connected user device, wherein: The user's hearing device extracts the speech features and sends them to the connected user device, whereby the connected user device calculates or updates the voiceprint model and sends it back to the hearing device; or The connected user equipment employs a mobile application that monitors the user's calls and / or other speaking activities and performs a speech feature extraction component attached to the voiceprint modeling.

6. The method of claim 1, wherein, In addition to the aforementioned speaker identification In the hearing system, one or more additional acoustic quality and / or personal communication conditions related to the wireless personal communication are monitored and / or analyzed; and The steps for automatically establishing, joining, and / or leaving wireless personal communication connections between the user's hearing device and the corresponding communication devices of other conversation participants also depend on additional conditions.

7. The method of claim 6, wherein, The additional conditions include: Environmental signal-to-noise ratio; and / or The existence of predefined environmental scenarios relating to the user and / or other people and / or surrounding objects and / or weather, wherein such scenarios can be identified by a corresponding classifier provided by the hearing device or the hearing system.

8. The method as described in claim 1, in, Once a wireless personal communication connection is established between the user's hearing device and the other speaker's communication device... The user's hearing device continuously monitors and analyzes the user's acoustic environment, and disconnects the wireless personal communication connection if the speaker's voiceprint is no longer identified within a predetermined time interval.

9. The method as described in claim 1, in, If a wireless personal communication connection is established between the user's hearing device and the communication devices of multiple other conversation participants, The user's hearing device then monitors and analyzes the user's acoustic environment, and, depending on at least one predetermined sorting criterion, disconnects wireless personal communication connections with some of these communication devices in order to form smaller conversation groups.

10. The method of claim 9, wherein, The at least one predetermined sorting criterion includes one or more of the following: Dialogue overlap; The directional gain is determined by the user's hearing device to characterize the orientation of the user's head relative to the orientation of the other participants in the conversation; The spatial distance between the user and the other corresponding dialogue participants.

11. The method of claim 1, further comprising: A user interface is presented to the user for notifying the user of other identified conversation participants and for establishing, joining, or leaving a wireless personal communication connection between the hearing device and one or more communication devices used by the one or more identified other conversation participants.

12. A computer program product for wireless personal communication using a hearing device, wherein, The computer program product includes a computer program, the hearing device is worn by a user and equipped with at least one microphone and a sound output device, and the computer program, when executed by a processor, is adapted to perform the steps of the method as described in any one of claims 1 to 11.

13. A computer-readable medium storing a computer program according to claim 12.

14. A hearing system comprising a hearing device worn by a user and optionally a connected user device, wherein, The hearing device includes: microphone; A processor for processing signals from the microphone; A sound output device for outputting a processed signal to the ear of the user of the hearing device; A transceiver for exchanging data with communication devices used by other dialogue participants and optionally with the connected user equipment; and The hearing system is adapted to perform the method as described in any one of claims 1 to 11.

Images