A model-based hearing compensation method and device

Abstract

This invention discloses a hearing compensation method and device based on model learning, the steps of which include: 1) using a neural network to perform nonlinear transformation on the sound signal S to obtain a compensation sound signal S for the target population with hearing impairment. NC ;2) Using a hearing impairment simulator to process the compensated sound signal S NC Nonlinear processing is performed to generate a nonlinear distortion signal S′; then S′ is used as the input to the neural network, S... NC Given the label data corresponding to S′, we obtain the training data (S′, S′). NC 1) Train the neural network; 2) Update the neural network in step 1) using the neural network trained in step 2); 3) Repeat steps 1) to 3) until the set termination condition is reached; 4) For a given sound signal, use the trained neural network to perform nonlinear transformation to generate compensating sound for playback to people with the target degree of hearing loss. This invention can more effectively compensate for hearing loss.

Description

Technical Field

[0001] This invention belongs to the field of hearing aid technology and relates to hearing compensation methods, specifically to a hearing compensation method and device based on model learning. Background Technology

[0002] Hearing loss is a significant public health issue. According to the World Health Organization's 2021 Global Hearing Assessment Report, approximately 1.5 billion people worldwide currently suffer from hearing loss, of whom 430 million require rehabilitation services. By 2050, this number is projected to increase to 2.5 billion, with 700 million requiring hearing rehabilitation. The diagnostic standard for hearing loss is the hearing threshold. The hearing threshold is determined by measuring the minimum sound pressure level at which a subject can correctly perceive a predetermined percentage of a pure tone signal played at a fixed frequency under specific environmental and equipment conditions. The hearing threshold reflects the degree of hearing loss at different frequencies. Normal hearing individuals have a bilateral hearing threshold of 25 dB HL or lower; a threshold higher than this indicates hearing loss. The most obvious characteristic of hearing loss is a decrease in audibility, which means previously audible signals are no longer audible. This is usually related to an increase in the neuronal firing threshold due to the loss of inner hair cells. Furthermore, hearing loss is accompanied by a loss of suprathreshold functions, such as reduced frequency selectivity, narrowed dynamic range of loudness perception, and weakened or absent input-output nonlinear compression characteristics, which are usually related to the loss of outer hair cells in the cochlea. Hearing loss causes communication barriers, affecting interpersonal relationships and work ability, leading to social isolation and a decline in quality of life. Studies have shown that compared to individuals without hearing loss, older adults with hearing loss have a higher risk of falls, dementia, depression, and death. Children with hearing loss experience delayed language development, which has a significant adverse impact on their academic performance. From a socioeconomic perspective, the World Health Organization estimates that hearing loss causes a global loss of $750 billion annually, including costs to the health sector, education support, and economic losses due to lost productivity. Hearing loss affects physical and mental health and incurs high socioeconomic costs, therefore early detection and intervention are necessary.

[0003] Providing appropriate hearing rehabilitation services can compensate for the hearing loss of the vast majority of patients, thereby improving their speech communication ability and quality of life. Hearing aids are the most commonly used hearing loss compensation device. They amplify external sounds and compensate for hearing loss to a certain extent. They typically employ a prescription-based hearing compensation method, which explicitly specifies the compensation parameters required for hearing compensation. The ultimate goal of hearing compensation is achieved by selecting and setting specific values ​​for these parameters. In practice, the audiologist usually pre-measures certain characteristics of the patient (such as hearing threshold), derives specific values ​​for the compensation parameters (such as hearing aid gain, compression threshold, compression range, compression ratio, etc.) based on the prescription formula, and then fine-tunes the compensation parameters based on the patient's subjective feedback. However, compared to a normal auditory system, auditory impairment causes non-linear distortion of sound. The essence of hearing compensation is to non-linearly enhance sound, but the compensation parameters set by the prescription formula may not accurately characterize this non-linearity required for hearing compensation. Secondly, different prescription formulas derive different compensation parameters, and choosing the appropriate prescription formula often depends on the experience of the audiologist. Finally, after completing the basic fitting using the prescription formula, the audiologist often needs to finely adjust the compensation parameters to meet the patient's personalized needs. This process often requires repeated adjustments and tests, which places high demands on the audiologist's skills. Summary of the Invention

[0004] To address the shortcomings of existing methods, this invention proposes a model-based hearing compensation method and device. This method aims to directly learn the nonlinear transformation corresponding to hearing compensation using an auditory model and a deep neural network. Compared to traditional hearing compensation methods, this approach eliminates the need to explicitly set various compensation parameters, resulting in lower operational complexity in subsequent fitting procedures. Furthermore, neural networks have a stronger ability to characterize nonlinear mappings, enabling more effective compensation for hearing loss. Therefore, this method is a more suitable hearing compensation approach.

[0005] The basic idea of ​​the model-based hearing compensation method proposed in this invention is as follows: Compared to a normal auditory system, an auditory impairment system causes nonlinear distortion of sound. By using a neural network to learn a certain nonlinear change in sound (i.e., the inverse change of the nonlinear distortion caused by the auditory impairment system), the compensated speech perceived by the hearing-impaired subject is made as consistent as possible with the original speech perceived by a normal person, thus effectively compensating for the patient's hearing loss. The key innovation of this invention lies in replacing the compensation module in traditional hearing compensation methods with a neural network that has a stronger ability to fit nonlinear changes; and introducing a hearing impairment simulator to simulate the nonlinear distortion of sound in the damaged auditory system, which is then used to guide the training of the neural network.

[0006] The technical solution of this invention is as follows:

[0007] A model-based hearing compensation method includes the following steps:

[0008] 1) Utilizing neural networks to replace the compensation module in traditional hearing compensation technology;

[0009] 2) Introduce a hearing loss simulator to simulate the nonlinear distortion of sound in a damaged auditory system;

[0010] 3) The neural network in step 1) and the hearing impairment simulator in step 2) are used together to generate the paired data (i.e., data-labels) required for training the neural network.

[0011] 4) Initialize the neural network in step 1) using the neural network trained in step 3), and repeat steps 1) to 3) until the set termination condition is reached. The termination condition is that the number of network iterations or the error reaches a limited value;

[0012] 5) For a given audio signal, the neural network trained in step 4) is used to process it to generate the compensatory sound to be played to the hearing-impaired subject.

[0013] Furthermore, the neural network described in step 1) is mainly used to generate the compensated sound signal. Specifically, the input to the network is the original sound signal, and the output is the sound signal after undergoing some nonlinear transformation (i.e., the compensated sound signal).

[0014] Furthermore, the hearing impairment simulator described in step 2) is mainly used to simulate the nonlinear distortion of sound in the damaged hearing system. Its purpose is to make the response excited by the processed signal in the normal hearing system consistent with the response excited in the damaged hearing system.

[0015] Furthermore, step 3) refers to the process of using the neural network in step 1) and the hearing impairment simulator in step 2) to jointly generate paired data for neural network training. This means that, given a sound signal S, the neural network described in step 1) is first used to process the signal and generate a sound signal S that has undergone a certain nonlinear transformation. NC (i.e., the sound signal after compensation by the current neural network); then, using the hearing loss simulator described in step 2) to test S NC After processing, a nonlinear distortion signal S′ is generated, which yields the paired data (S′, S′) used for neural network training. NC ), where S NC This is the label corresponding to S′. For people with normal hearing, S′ and S′ are... NC They are the same.

[0016] Furthermore, the neural network structure trained in step 3) is the same as the neural network structure and network weights described in step 1).

[0017] A hearing compensation device based on model learning, characterized in that it includes a hearing loss simulator, a paired data generation module, and a hearing compensation module; wherein...

[0018] The hearing loss simulator is used to process sound signals so that the processed signal elicits a response in the normal auditory system that is consistent with the response elicited by the unprocessed stimulus in the damaged auditory system.

[0019] The pairing data generation module is used to generate pairing data for neural network training at the current moment using the hearing impairment simulator and the neural network at the previous moment.

[0020] The hearing compensation module is used to process sound using a trained neural network to generate a signal that has undergone a certain nonlinear transformation (i.e., the compensated signal).

[0021] Compared with the prior art, the positive effects of the present invention are as follows:

[0022] This invention uses a hearing loss simulator and a deep neural network to directly learn the nonlinear transformation corresponding to hearing compensation. Compared with traditional prescription-based hearing compensation methods, this method eliminates the need to explicitly set various compensation parameters, reducing the complexity of subsequent fitting procedures. Furthermore, neural networks have a stronger ability to characterize nonlinear mappings, enabling more effective compensation for hearing loss. Therefore, this invention can serve as a feasible solution for hearing compensation. Attached Figure Description

[0023] Figure 1 This is a basic framework diagram for model-based hearing compensation.

[0024] Figure 2 This patent proposes a self-supervised learning framework.

[0025] Figure 3 This is a typical audiogram of declining hearing loss (moderate hearing loss).

[0026] Figure 4 This is the result of the experimental evaluation. Detailed Implementation

[0027] The specific implementation details of the present invention will be described in more detail below. Figure 1 This is a framework diagram of the model-based hearing compensation method proposed in this invention. The specific implementation steps of this invention include neural network compensation, a hearing impairment simulator, and neural network training. The specific implementation process of each step is as follows:

[0028] 1. Neural network compensation

[0029] This method uses a neural network to replace the compensation module in traditional hearing compensation techniques. The sound S is input into the corresponding DNN, which performs a non-linear transformation on the sound and outputs a compensated signal S. NC ,Right now

[0030] S NC (n)=DNN(S(n))

[0031] The network model structure of DNN can be found in [reference needed]. Figure 2 .

[0032] 2. Hearing Impairment Simulator

[0033] The hearing impairment simulator introduced in this method is mainly used for nonlinear processing of sound, so that the response of the processed sound in the normal auditory system is consistent with the response of the unprocessed stimulus in the damaged auditory system. The hearing impairment simulator processes S1 into S2; where S1 is a sound with a stimulus level of H for the target hearing impairment population, and S2 is a sound with a stimulus level of H for the normal hearing population. In this patent, characteristic loudness (a loudness density, usually related to neural activity at the corresponding characteristic frequency) is selected as the response of sound in the auditory system, and a loudness model suitable for hearing impairment patients is used to set up the hearing impairment simulator. With the damage of the inner and outer hair cells in the cochlea, loudness perception may be affected by the following changes: 1) threshold elevation; 2) weakening or disappearance of the nonlinear compression characteristics of the basilar membrane; 3) weakening or disappearance of frequency selectivity. Loudness models usually assume a threshold HL at a specific frequency. Total It can be expressed in the following form

[0034] HL OHC +HL IHC =HL TOTAL

[0035] HL OHC and HL IHC These represent the hearing threshold elevation caused by damage to outer hair cells and inner hair cells, respectively. Therefore, the effect of outer hair cell damage on threshold elevation can be adjusted by adjusting the ratio R. OHC To depict:

[0036] R OHC =HL OHC / HL TOTAL

[0037] The hearing impairment simulator processes sound in the following way: First, a short-time Fourier transform is performed on the sound signal to obtain the power spectrum and phase spectrum of each frame. Then, the power spectrum is multiplied by an intensity- and frequency-dependent attenuation factor to obtain the processed power spectrum. Finally, the processed power spectrum and the original phase spectrum are subjected to a short-time inverse Fourier transform to obtain the processed signal. The attenuation factor is calculated in the following way, the purpose of which is to ensure that the characteristic loudness of the processed stimulus perceived in the normal ear is consistent with the loudness of the unprocessed stimulus perceived in the damaged ear. First, the excitation level E is calculated based on the power spectrum of the sound signal and the auditory filter.

[0038] E=WP

[0039] Where W describes the amplitude-frequency response of the auditory filter, and P is the power spectrum of a given stimulus. Then, the characteristic loudness N of the given stimulus perceived by the damaged ear is calculated. HI ′, and sequentially calculate the excitation level E required to achieve the same loudness characteristic in a normal ear. SIM Specifically, the characteristic loudness N′ in a normal ear NH The following formula is used for calculation:

[0040]

[0041] Where C is a constant (0.047), E THRQ This represents the maximum excitation level produced by a sinusoidal signal at the normal ear threshold. G is a variable related to the cochlear amplifier gain. A and α are variables dependent on G, and their values ​​and relationships can be found in ISO 532-2:2017.

[0042] Characteristic loudness N′ perceived by the damaged ear HI The following formula is used for calculation:

[0043]

[0044] Compared to normal hearing, damage to the outer hair cells will reduce the gain of the cochlear amplifier from G to [missing value]. This affects the values ​​of A and α (for clarity, A is used in the formula). HI and α HI (This indicates that) damage to the inner hair cells will reduce the excitation level E to [a certain value]. Therefore, once the excitation level E in the damaged ear and the hearing threshold elevation (HL) caused by damage to the outer hair cells and inner hair cells are given, OHC and HL IHC ), HL OHC and HL IHC Given a specific value, the characteristic loudness N′ perceived by the damaged ear can be calculated.HI Then, the excitation level E required to achieve the same loudness characteristic in a normal ear is calculated. SIM Because we expect the response (N′) elicited by the unprocessed stimulus (E) in the damaged auditory system. HI ) and post-treatment stimulus (E) SIM The response elicited in the normal auditory system (i.e., N′) NH ) are consistent; therefore N′ NH =N′ HI Then, according to N′ NH The formula (since it is a normal auditory system, at this time HL) OHC With HL IHC Since both are 0, we can deduce E. SIM .

[0045] The attenuation factor D used in the hearing loss simulator is defined as follows:

[0046] D = E SIM / E.

[0047] 3. Neural Network Training

[0048] The basic idea of ​​this invention is to introduce an auditory model to simulate the sound processing of a hearing-impaired system and guide the neural network to learn a certain nonlinear change in sound (i.e., the inverse change of the nonlinear distortion caused by the hearing-impaired system). This ensures that the compensated sound perceived by the hearing-impaired subject is as consistent as possible with the original sound perceived by a normal person; that is, the sound signal generated after the original sound passes through the neural network and the auditory model sequentially is consistent with the original sound. However, using the original speech as supervision, the gradient cannot be backpropagated to the neural network through the auditory model. Therefore, this method uses the compensated speech generated by the neural network trained in the previous time step as supervision, and uses the sound signal of the compensated speech passing through the auditory model as the input to train the neural network.

[0049] Specifically, such as Figure 2 As shown, given a sound signal S, the neural network DNN trained in the previous time step is first used... t-1 (Initial network random initialization) process generates a sound signal S that has undergone some nonlinear transformation. NC (i.e., the sound signal after compensation by the current neural network); then the auditory model is used to analyze S. NC After processing, a nonlinear distortion signal S′ after processing by the hearing impairment system is generated, and the paired data (S′, S) can be obtained. NC ), where S′ is the input of the neural network, S NC The label data corresponding to S'; the paired data is used to train a DNN neural network. t Finally, the DNN trained at the current time step is used.t Update DNN t-1 Repeat the above process until the final termination condition is met (i.e., the number of iterations reaches the set value or the error decreases to the set value).

[0050] The advantages of the present invention will be explained below with reference to specific embodiments.

[0051] To verify the effectiveness of the hearing compensation proposed in this invention, we recruited 16 participants with normal hearing for evaluation. To simulate hearing loss, the stimuli were processed using a hearing loss simulator before being played to the participants. The experiment included three experimental conditions (no compensation, compensation using the method proposed in this patent, and compensation using the gain calculated by the traditional prescription formula CAM2) and one reference condition (normal speech was played directly to the participants with normal hearing). These four conditions were combined in pairs, resulting in six possible combinations. In the test, one combination was randomly selected, and the sound was processed. After hearing two speech segments corresponding to different conditions, the participants had to choose which condition's speech sounded clearer and more natural.

[0052] Figure 3 The experimental results are shown. The results indicate that the hearing compensation method proposed in this patent is significantly more effective than the uncompensated condition and traditional methods, and shows no significant difference from the reference condition. This demonstrates that the hearing compensation method proposed in this invention can learn a certain nonlinear transformation of sound, thus compensating for the hearing loss of patients with hearing impairment to a certain extent.

[0053] Although specific embodiments and accompanying drawings of the invention have been disclosed for illustrative purposes to aid in understanding and implementing the invention, those skilled in the art will understand that various substitutions, variations, and modifications are possible without departing from the spirit and scope of the invention and the appended claims. Therefore, the invention should not be limited to the content disclosed in the preferred embodiments and accompanying drawings.

Claims

1. A hearing compensation method based on model learning, comprising the following steps: 1) By using a neural network to perform nonlinear transformation on the sound signal S, a compensating sound signal Sc is obtained for the target population with varying degrees of hearing impairment. NC ; 2) The compensation sound signal S is processed using a hearing impairment simulator. NC Nonlinear processing is performed to generate a nonlinear distortion signal S′; then S′ is used as the input to the neural network, S NC Given the label data corresponding to S′, we obtain the training data (S′, S′). NC The neural network is trained by the hearing impairment simulator, which first calculates the excitation level E = WP based on the power spectrum P of the sound signal and the amplitude-frequency response W of the auditory filter; then it calculates the characteristic loudness N of a given stimulus perceived in the damaged ear. HI ′, and sequentially calculate the excitation level E required to achieve the same loudness characteristic in a normal ear. SIM Then, the attenuation factor D=E used in the hearing impairment simulator is calculated. SIM / E; Then, a short-time Fourier transform is performed on the audio signal to obtain the power spectrum and phase spectrum of each frame of the signal. Then, the power spectrum of the signal is multiplied by the attenuation factor to obtain the processed power spectrum. Finally, the processed power spectrum and the original phase spectrum are subjected to a short-time inverse Fourier transform to obtain the processed signal. 3) Update the neural network in step 1) using the neural network trained in step 2); 4) Repeat steps 1) to 3) until the set termination condition is met; 5) For a given audio signal, the neural network obtained after training in step 4) is subjected to nonlinear transformation to generate compensating sound for the target group of people with hearing loss.

2. The method as described in claim 1, characterized in that, The neural network is a DNN.

3. The method as described in claim 1, characterized in that, The termination condition is when the number of training iterations or the error reaches a certain limit.

4. A hearing compensation device based on model learning, characterized in that, It includes a hearing loss simulator, a paired data generation module, and a hearing compensation module; among which The hearing loss simulator is used to process sound signals so that the processed signal elicits a response in the normal auditory system consistent with the response elicited by the unprocessed stimulus in the damaged auditory system. The simulator first calculates the excitation level E = WP based on the power spectrum P of the sound signal and the amplitude-frequency response W of the auditory filter; then it calculates the characteristic loudness N of a given stimulus perceived by the damaged ear. HI ′, and sequentially calculate the excitation level E required to achieve the same loudness characteristic in a normal ear. SIM Then, the attenuation factor D=E used in the hearing impairment simulator is calculated. SIM / E; Then, a short-time Fourier transform is performed on the audio signal to obtain the power spectrum and phase spectrum of each frame of the signal. Then, the power spectrum of the signal is multiplied by the attenuation factor to obtain the processed power spectrum. Finally, the processed power spectrum and the original phase spectrum are subjected to a short-time inverse Fourier transform to obtain the processed signal. The hearing compensation module is used to perform nonlinear transformation on the sound signal S using a neural network to obtain a compensation sound signal S tailored to the target population with varying degrees of hearing impairment. NC ; The pairing data generation module is used to generate the compensation sound signal S using a hearing impairment simulator. NC Nonlinear processing is performed to generate a nonlinear distortion signal S′; then S′ is used as the input to the neural network, S NC Given the label data corresponding to S′, we obtain the training data (S′, S′). NC Train the neural network.

Images