Speech enhancement method, device and hearing aid based on dual-channel neural network time-frequency masking

Abstract

The present invention relates to the technical field of hearing aid speech enhancement, and in particular to a speech enhancement method, device and hearing aid equipment based on dual-channel neural network time-frequency masking. The method comprises the following steps: using two microphones to receive speech signals under noise and reverberation conditions, performing preliminary speech enhancement on each microphone signal received through a trained single-channel neural network, and removing noise in the same direction as the target speech; different from traditional positioning that blindly relies on signal energy, an adaptive ratio mask provides a view of the target signal, can accurately identify the speech-dominant time-frequency unit on each microphone channel, and uses the time-frequency unit for sound source positioning, thereby ensuring high positioning accuracy under noise and reverberation conditions; and removing noisy speech in different directions from the target speech and suppressing room reverberation through the calculated weight input WPD beamformer, thereby obtaining enhanced speech with good speech quality and high intelligibility.

Description

Technical Field

[0001] The present invention relates to the technical field of hearing aid speech enhancement, and in particular to a speech enhancement method, device and hearing aid equipment based on dual-channel neural network time-frequency masking. Background Art

[0002] Due to the damage to the auditory organs, the speech recognition ability of hearing loss patients decreases, making their speech comprehension in noisy environments much lower than that of people with normal hearing. Hearing loss can cause great inconvenience to life. According to the National Health Survey, as people are more exposed to portable music players, temporary and permanent hearing loss phenomena are becoming younger. Wearing hearing aids is one of the safest and most effective ways to solve the problem of hearing loss, and it plays an irreplaceable role in improving the hearing of people with hearing loss. The speech comprehension of hearing loss patients in noisy environments is one of the key factors affecting the use of hearing aids. Therefore, speech enhancement technology is a core technology in hearing aids. It can effectively eliminate background noise and improve speech quality, thereby improving patients' speech comprehension in noisy environments.

[0003] Common hearing aid speech enhancement algorithms can be divided into two categories, single-channel speech enhancement and microphone array speech enhancement, according to the number of microphones at the receiving end. Single-channel speech enhancement includes spectral subtraction, Wiener filtering, single-channel neural network speech enhancement, etc., which uses the statistical information of speech or noise for denoising. It only needs a single microphone to achieve low cost; but since the single-channel speech enhancement algorithm can only obtain a small amount of speech information and cannot use the spatial information of the sound, its performance is limited. Digital hearing aids generally use speech enhancement algorithms based on microphone arrays, which not only use the time-frequency information of speech but also consider the spatial information of the sound source. It can effectively extract sound signals within a certain spatial range, so that the hearing aid can provide good speech understanding in a high signal-to-noise ratio background; however, the auditory experience in a low signal-to-noise ratio environment is not as satisfactory as expected.

[0004] In recent years, several effective multi-channel hearing aid speech enhancement algorithms have been proposed. C. Knapp and G. Carter proposed a generalized cross correlation matrix with phase transform (Generalized Cross Correlation with Phase Transform, GCC-PHAT). The algorithm can reduce noise, but the sum of the GCC-PHAT coefficients shows false peaks in interference sources or reverberant environments, resulting in inaccurate estimated direction of arrival (DOA) and poor noise reduction performance. R. Schmidt et al. proposed a multiple signal classification algorithm, which uses the start frame based on unreasonable assumptions or the silent frame predicted by voice activity detection to estimate the covariance matrix of the noise. The noise reduction performance is limited by the number of microphones. MLSeltzer proposed a post-filtering algorithm, which can effectively remove noise by performing spatial filtering through phase information, but it is often impossible to achieve high-quality speech noise reduction in a reverberant multi-source environment.

[0005] With the development of deep learning, a microphone array speech enhancement algorithm based on deep learning has been proposed, which effectively improves the quality of enhanced speech. J. Woodruff used interaural time as a spatial feature input for neural network training, and the clarity of reconstructed speech was significantly improved. Martin Raspaud et al. input interaural time difference, inter-channel phase difference and speech spectrum to train deep autoencoders for enhancement. However, these algorithms all assume that the target speech comes from the front, so the enhancement effect is poor when the target speech comes from other directions. And due to the hardware limitations of digital hearing aids, only two to three low-complexity microphone arrays can be used for preliminary speech enhancement, and the reconstructed speech has low intelligibility.

[0006] In addition, the traditional WPD-based convolutional beamformer method for hearing aid speech enhancement calculates the arrival time difference through methods such as GCC-PHAT, and then estimates the direction vector in combination with the geometric assumptions of the microphone array; however, the above traditional methods cannot accurately calculate the statistical values ​​of noise and speech that are critical to beamforming. In the presence of interference conditions such as noise and reverberation, the array cannot accurately form a beam for the desired signal and form a null in the interference direction. Noise is ever-changing, and existing methods can only remove noise in different directions from the target speech. Some methods introduce new noise while removing noise, and some methods cause irreversible damage to the speech after removing noise. Therefore, in practice, the speech enhancement effect is poor, there is a lot of residual noise, and the speech intelligibility is low. The enhancement effect of the beamforming algorithm depends largely on the number and geometry of the microphone array. Due to hardware limitations, digital hearing aids can only use two to three microphones, which limits the performance of the algorithm. Summary of the invention

[0007] In view of the technical defects and technical drawbacks existing in the prior art, the embodiments of the present invention provide a speech enhancement method, device and hearing aid device based on dual-channel neural network time-frequency masking that overcome the above problems or at least partially solve the above problems, and solve the problems that the performance of the existing hearing aid speech enhancement algorithm drops sharply in a noisy environment, the ability to suppress directional noise is poor, and the enhanced speech has low intelligibility due to the failure to simultaneously consider room reverberation.

[0008] As one aspect of an embodiment of the present invention, a speech enhancement method based on dual-channel neural network time-frequency masking is provided, the method comprising:

[0009] Train a neural network to estimate an adaptive ratio mask to perform monaural speech enhancement on each microphone signal and remove background noise in the same direction as the target speech.

[0010] Identifying the target speech time-frequency unit and the noise time-frequency unit by the adaptive ratio mask, and calculating the mask weight representing the importance of the time-frequency unit to the DOA estimation;

[0011] Combined with the spatial covariance matrix, the speech covariance matrix is ​​calculated by the mask weights of the target speech time-frequency unit, and the noise covariance matrix is ​​calculated by the mask weights of the noise time-frequency unit. The direction vector of the WPD beamformer is calculated through the speech covariance matrix to locate the target speech.

[0012] The noise covariance matrix and the direction vector estimated by the speech covariance matrix are combined to calculate the weight of the WPD beamformer. The beamformer removes noise in different directions from the target speech and suppresses reverberation to obtain the final enhanced speech.

[0013] Furthermore, the method of “training a single-channel neural network to estimate an adaptive ratio mask” includes:

[0014] The signal-to-noise ratio is used to construct a proportional factor, and an adaptive ratio mask is established to adaptively adjust the ratio between the ideal binary mask and the ideal ratio mask according to the signal-to-noise ratio of the time-frequency unit; the direct speech signal is taken as the target, and the remaining components are noise, and the mask value of each microphone signal is calculated separately.

[0015] Further, the adaptive ratio mask is:

[0016] AM i (t,f)=(1-α(t,f))·IBM i (t,f)+α(t,f)·IRM i (t,f)

[0017] Where: IBM represents the ideal binary mask, IRM represents the ideal ratio mask;

[0018]

[0019]

[0020]

[0021]

[0022] Where: E d (t,f) represents the energy of the direct signal of the i-th microphone, E h (t,f)+E n (t,f) represents the energy of the i-th microphone noise; SNR(t,f) represents the signal-to-noise ratio of the time-frequency unit, S 2 (t,f),N 2 (t, f) represent the energy of speech and noise respectively; λ and β are used to adjust the shape of the exponential function; through experimental verification, their values ​​are set to -5 and 2.

[0023] Furthermore, the training method of the "neural network" includes:

[0024] With the goal of minimizing the mean square error function, the multi-resolution auditory cepstral coefficient features and adaptive ratio mask of the speech in the training set are extracted as the input of the model. Both the input and output are normalized by mean variance, and the neural network model is trained and saved.

[0025] The multi-resolution auditory cepstral coefficient feature includes four cochlear representations of different resolutions, and uses a nonlinear power function that conforms to human auditory perception to compress signal energy and discrete cosine transform decorrelation.

[0026] Furthermore, the neural network model includes a 4-layer structure, including two hidden layers, each hidden layer is provided with 1024 nodes, and the output layer is provided with 64 nodes. The hidden layer uses a linear rectification activation function, and the output layer uses a Sigmoid activation function.

[0027] Furthermore, the method further comprises:

[0028] Using the adaptive ratio mask of each microphone signal, the mask weights η(t,f) and ξ(t,f) representing the importance of each time-frequency unit to DOA estimation are calculated; where:

[0029]

[0030]

[0031] i represents the i-th microphone signal, D represents the total number of microphone signals, D=2;

[0032] Combining the adaptive ratio mask and the power-normalized spatial covariance matrix R with prediction delay, the ratio mask weighted noise covariance matrix τ and speech covariance matrix α are obtained;

[0033]

[0034]

[0035] Estimating the direction vector using the speech covariance matrix

[0036]

[0037] Where P{g} represents the calculated principal eigenvector;

[0038] Given the time-varying power and direction vector required by the signal, calculate the weight ω of the WPD beamformer,

[0039]

[0040] Get enhanced signal

[0041] Furthermore, the method further comprises:

[0042] Using dual microphones to receive a speech signal containing noise and reverberation, and performing a short-time Fourier transform on the speech signal to obtain a noisy speech spectrum, wherein the noisy speech spectrum includes an amplitude spectrum and a phase spectrum;

[0043] The amplitude spectrum of each microphone signal is input into a single-channel neural network to estimate the adaptive ratio mask and preliminary speech enhancement;

[0044] Multiply the adaptive ratio masks of each microphone signal and calculate the mask weight representing the importance of each time-frequency unit to DOA estimation;

[0045] Combining the mask weights and a power-normalized speech signal spatial covariance matrix with prediction delay to obtain a ratio-masked weighted noise covariance matrix and a speech covariance matrix;

[0046] Calculate an estimate of the direction vector using the speech covariance matrix;

[0047] The weight of the WPD beamformer is calculated using the estimated values ​​of the noise cosquare matrix and the direction vector, and the weight is multiplied by the amplitude spectrum to obtain the amplitude spectrum of the enhanced speech;

[0048] The final enhanced speech is obtained by combining the phase spectrum of the noisy speech and the amplitude spectrum of the enhanced speech.

[0049] As another aspect of the embodiment of the present invention, a speech enhancement device based on dual-channel neural network time-frequency masking is provided, the device comprising:

[0050] Preliminary enhancement module: used to train the neural network to estimate the adaptive ratio mask, perform monaural speech enhancement on each microphone signal, and remove background noise in the same direction as the target speech;

[0051] Target positioning module: used to identify the target speech time-frequency unit and the noise time-frequency unit through the adaptive ratio mask, calculate the mask weight representing the importance of the time-frequency unit to DOA estimation; combine the spatial covariance matrix, calculate the speech covariance matrix from the mask weight of the target speech time-frequency unit, calculate the noise covariance matrix from the mask weight of the noise time-frequency unit, and estimate the direction vector required by the beamformer through the speech covariance matrix to locate the target speech;

[0052] The speech enhancement module is used to combine the noise covariance matrix and the estimated direction vector, remove the noise in different directions from the target speech and suppress the reverberation through the WPD beamformer, and obtain the final enhanced speech. As another aspect of an embodiment of the present invention, a hearing aid device is provided, including: a processor and a memory, the memory is used to store computer program code, the computer program code includes computer instructions, when the processor executes the computer instructions, the hearing aid device executes any of the above-mentioned speech enhancement methods based on dual-channel neural network time-frequency masking.

[0053] As another aspect of an embodiment of the present invention, a storage medium is provided, in which a computer program is stored. The computer program includes program instructions, and when the program instructions are executed by a processor of a terminal device, the processor executes any of the above-mentioned speech enhancement methods based on dual-channel neural network time-frequency masking.

[0054] The embodiments of the present invention achieve at least the following technical effects:

[0055] The embodiment of the present invention provides a hearing aid speech enhancer based on dual-channel neural network time-frequency masking that integrates sound source localization, speech denoising, and dereverberation; it combines single-microphone and multi-microphone processing for speech enhancement, solving the problem of poor enhancement effect of traditional hearing aids due to hardware limitations. In single-microphone processing, the neural network model preliminarily enhances the signal of each microphone and estimates an adaptive ratio mask. Due to its powerful nonlinear mapping ability, the neural network can better suppress background noise and remove noise in the same direction as the target speech. The learning target adaptive mask combines the advantages of binary masks and soft masks to more accurately identify speech-dominated time-frequency units and reconstruct speech with high intelligibility.

[0056] In addition, the traditional beamformer based on directional vector positioning estimates the directional vector through DOA estimation and the geometry of the microphone array; when the signal-to-noise ratio is low, it is difficult to obtain an accurate DOA estimate. Unlike traditional methods, the embodiment of the present invention combines an adaptive ratio mask with a directional vector positioning method estimated from the main eigenvector of the speech covariance matrix. The adaptive mask guides the directional vector positioning method, only using the speech-dominated time-frequency units for positioning, accurately calculating the covariance matrix of speech and noise, suppressing the effects of interference and reverberation on positioning, and forming an accurate beam in the target speech direction. Compared with other algorithms, the algorithm significantly improves the speech enhancement effect at different signal-to-noise ratios, and the speech intelligibility is also significantly improved.

[0057] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structure described in the written description, claims, and drawings.

[0058] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0060] Figure 1 This is a flow chart of a speech enhancement method based on dual-channel neural network time-frequency masking according to an embodiment of the present invention;

[0061] Figure 2 A schematic diagram of a neural network training and testing process according to an embodiment of the present invention;

[0062] Figure 3 A schematic diagram of a vector positioning method according to an embodiment of the present invention;

[0063] Figure 4 A schematic diagram of a dual-channel neural network speech enhancement method according to an embodiment of the present invention;

[0064] Figure 5 A flow chart of speech enhancement using a speech enhancement method based on dual-channel neural network time-frequency masking according to an embodiment of the present invention;

[0065] Figure 6 A schematic diagram of a test environment according to an embodiment of the present invention;

[0066] Figure 7 A visualization of the ideal IRM and the estimated IRM tested for an embodiment of the present invention;

[0067] Figure 8 A visualization diagram of the ideal AM and the estimated AM tested for an embodiment of the present invention;

[0068] Fig. 9 This is a comparison chart of average values ​​of enhancement effects tested in one embodiment of the present invention;

[0069] Fig.10 A time domain waveform diagram showing the comparison of enhancement effects tested in an embodiment of the present invention;

[0070] Fig.11 A spectrogram comparing the enhancement effects tested in one embodiment of the present invention. DETAILED DESCRIPTION

[0071] In order to explain the technical content, achieved objectives and effects of the present invention in detail, the following is an explanation in combination with the implementation modes and the accompanying drawings.

[0072] The accompanying drawings and the following description describe the optional embodiments of the present invention to teach those skilled in the art how to implement and reproduce the present invention. In order to teach the technical solution of the present invention, some conventional aspects have been simplified or omitted. Those skilled in the art will understand that variations or replacements derived from these embodiments will fall within the scope of protection of the present invention. Those skilled in the art will understand that the following features can be combined in various ways to form multiple variations of the present invention. Thus, the present invention is not limited to the following optional embodiments, but is only limited by the claims and their equivalents.

[0073] In one embodiment, a speech enhancement method based on dual-channel neural network time-frequency masking is provided. Figure 1 , Figure 2 , Figure 3 and Figure 4 As shown, the method includes:

[0074] S11 trains a neural network to estimate an adaptive ratio mask, performs monaural speech enhancement on each microphone signal, and removes background noise in the same direction as the target speech;

[0075] S12 identifies the target speech time-frequency unit and the noise time-frequency unit through the adaptive ratio mask, and calculates the mask weight representing the importance of the time-frequency unit to the DOA estimation;

[0076] S13 combines the spatial covariance matrix, calculates the speech covariance matrix by the mask weights of the target speech time-frequency unit, calculates the noise covariance matrix by the mask weights of the noise time-frequency unit, calculates the direction vector of the WPD beamformer through the speech covariance matrix, and locates the target speech.

[0077] S14 combines the noise covariance matrix and the direction vector estimated by the speech covariance matrix to calculate the weight of the WPD beamformer, removes noise in different directions from the target speech and suppresses reverberation, and obtains the final enhanced speech.

[0078] In this embodiment, a neural network is first trained to estimate an adaptive ratio mask of pure speech, and preliminary monaural speech enhancement is performed on each microphone signal to remove noise in the same direction as the target speech and improve speech clarity. Then, a directional vector localization method based on the adaptive ratio mask is used to estimate the potential of the arrival direction in a noisy reverberant environment, identify speech-dominant time-frequency units containing relatively pure phases to estimate the arrival direction, and locate the target sound source. Finally, the preliminary enhanced microphone signal and the estimated directional vector are input into a weighted power minimization distortionless response (WPD) convolutional beamformer to further remove noise in different directions from the target speech and suppress reverberation, thereby obtaining a hearing aid speech enhancer with high speech intelligibility.

[0079] In this embodiment, speech enhancement can be performed by estimating an adaptive ratio mask through a single-channel neural network to remove background noise in the same direction as the target speech. Then, the estimated adaptive ratio mask is used to accurately identify whether each time-frequency unit is dominated by the target speech or noise, and the covariance matrix of the speech is calculated from the speech-dominated time-frequency unit, and the covariance matrix of the noise is calculated from the noise-dominated time-frequency unit. An accurate beam is formed for the target speech to remove noise in different directions from the target speech. Compared with the traditional hearing aid speech enhancement algorithm, the method of this embodiment eliminates both background noise in the same direction as the speech and noise interference in different directions from the speech, and obtains enhanced speech with high speech intelligibility and clarity.

[0080] In one embodiment, the method of “training a single-channel neural network to estimate an adaptive ratio mask” includes:

[0081] The signal-to-noise ratio is used to construct a proportional factor, and an adaptive ratio mask is established to adaptively adjust the ratio between the ideal binary mask and the ideal ratio mask according to the signal-to-noise ratio of the time-frequency unit; the direct speech signal is taken as the target, and the remaining components are noise, and the mask value of each microphone signal is calculated separately.

[0082] In this embodiment, a new time-frequency mask, namely an adaptive ratio mask, is proposed. The mask combines the advantages of the traditional ideal ratio mask (Ideal Ratio Mask, IRM) and the ideal binary mask (Ideal Binary Mask, IBM), and can more accurately locate the time-frequency unit dominated by speech. The neural network model is trained to estimate the adaptive mask, which is used to preliminarily enhance the unprocessed signal and accurately estimate the direction vector required for beamforming. AM uses the signal-to-noise ratio to construct a proportional factor, and adaptively adjusts the proportion between the traditional ideal binary mask (Ideal Binary Mask, IBM) and the ideal ratio mask (IRM) according to the signal-to-noise ratio of the time-frequency unit, giving play to the respective advantages of the two masks. At low signal-to-noise ratios, the AM mask value is close to IRM, which can avoid IBM from eliminating interference while also shielding the effective signal. At high signal-to-noise ratios, the AM mask is closer to IBM, well maintaining the structure of the speech signal, more accurately identifying each time-frequency unit as dominated by speech or noise, and improving the quality of reconstructed speech. When defining the adaptive mask, the direct speech signal is taken as the target, the remaining components are noise, and the mask value of each microphone signal is calculated separately.

[0083] Further, the adaptive ratio mask is:

[0084] AM i (t,f)=(1-α(t,f))·IBM i (t,f)+α(t,f)·IRM i (t,f)

[0085] Where: IBM represents the ideal binary mask, IRM represents the ideal ratio mask;

[0086]

[0087]

[0088]

[0089]

[0090] Where: E d (t,f) represents the energy of the direct signal of the i-th microphone, E h (t,f)+E n (t,f) represents the energy of the i-th microphone noise; SNR(t,f) represents the signal-to-noise ratio of the time-frequency unit, S 2 (t,f),N 2 (t, f) represent the energy of speech and noise respectively; λ and β are used to adjust the shape of the exponential function; through experimental verification, their values ​​are set to -5 and 2.

[0091] The adaptive ratio mask in this embodiment can be used to process single microphone and multiple microphones differently. In single microphone processing, the neural network model preliminarily enhances each microphone signal and estimates the adaptive ratio mask. In multi-microphone processing, the adaptive ratio mask weighted covariance matrix is ​​used to estimate the direction vector, and the speech-dominated time-frequency (TF) unit used for positioning on each microphone channel is identified, so that accurate DOA estimation can be obtained in strong noise and reverberation environments. Finally, speech enhancement is performed through a convolution beamformer based on the improved WPD optimization criterion, so that the denoising and reverberation suppression effects are optimized at the same time, improving the performance of the hearing aid in a noisy environment.

[0092] In one embodiment, the training method of the "neural network" includes:

[0093] With the goal of minimizing the mean square error function, the multi-resolution auditory cepstral coefficient features and adaptive ratio mask of the speech in the training set are extracted as the input of the model. Both the input and output are normalized by mean variance, and the neural network model is trained and saved.

[0094] The multi-resolution auditory cepstral coefficient feature includes four cochlear representations of different resolutions, and uses a nonlinear power function that conforms to human auditory perception to compress signal energy and discrete cosine transform decorrelation.

[0095] In this embodiment, the single-channel neural network has a strong nonlinear mapping ability and a good ability to suppress non-stationary noise. This embodiment uses a nonlinear power function that is more in line with human auditory perception to compress signal energy, and discrete cosine transform (DCT) decorrelation to reduce the amount of calculation. Subsequently, the first-order and second-order differential derivatives of the MRACC features are calculated to capture the transient information of speech, and the dynamic and static combination comprehensively represents the nonlinear structure of speech to improve the quality of reconstructed speech. The learning target AM combines the advantages of IBM and IRM, and adaptively adjusts the mask value according to the signal-to-noise ratio of the time-frequency unit to improve the perception of noisy speech; moreover, the stability of the training process is maintained through mean-variance normalization.

[0096] In one embodiment, the neural network model includes a 4-layer structure, including two hidden layers, each hidden layer is provided with 1024 nodes, and the output layer is provided with 64 nodes. The hidden layer uses a linear rectification activation function, and the output layer uses a Sigmoid activation function.

[0097] In this embodiment, an optimal neural network is provided, which has a fast operation speed and a robust network. GreatIn the test phase, the normalized MRACC features of the test sample set are extracted and input into the trained neural network model to predict AM. Finally, the phase of the noisy speech is combined to reconstruct the speech and the best enhancement result is obtained.

[0098] In one embodiment, the method further comprises:

[0099] Using the adaptive ratio mask of each microphone signal, the mask weights η(t,f) and ξ(t,f) representing the importance of each time-frequency unit to DOA estimation are calculated; where:

[0100]

[0101]

[0102] i represents the i-th microphone signal, D represents the total number of microphone signals, D=2;

[0103] Combining the adaptive ratio mask and the power-normalized spatial covariance matrix R with prediction delay, the ratio mask weighted noise covariance matrix τ and speech covariance matrix α are obtained;

[0104]

[0105]

[0106] Estimating the direction vector using the speech covariance matrix

[0107]

[0108] Where P{g} represents the calculated principal eigenvector;

[0109] Given the time-varying power and direction vector required by the signal, calculate the weight ω of the WPD beamformer,

[0110]

[0111] Get enhanced signal

[0112] In this embodiment, a positioning method based on adaptive mask direction vector is proposed, the main idea of ​​which is to use the adaptive mask estimated by the neural network to identify the time-frequency TF units dominated by speech and noise, calculate the speech covariance matrix from the speech-dominated TF unit, calculate the noise covariance matrix from the noise-dominated TF unit, and accurately calculate the spatial covariance matrix and direction vector that are crucial for beamforming. The time-frequency unit dominated by the target speech carries clearer positioning information, and the adaptive mask gives more weight to the target speech unit, so that the direction vector contains enough information about the direction of the target speech.

[0113] In one embodiment, Figure 5 As shown, the method also includes:

[0114] S21 uses dual microphones to receive a speech signal containing noise and reverberation, and performs a short-time Fourier transform on the speech signal to obtain a noisy speech spectrum, wherein the noisy speech spectrum includes an amplitude spectrum and a phase spectrum;

[0115] S22 inputs the amplitude spectrum of each microphone signal into a single-channel neural network to estimate an adaptive ratio mask and preliminary speech enhancement;

[0116] S23 multiplies the adaptive ratio mask of each microphone signal and calculates a mask weight representing the importance of each time-frequency unit to the DOA estimation;

[0117] S24 combines the mask weights and the power-normalized speech signal space covariance matrix with prediction delay to obtain a ratio mask-weighted noise covariance matrix and a speech covariance matrix;

[0118] S25 calculates an estimated value of the direction vector through the speech covariance matrix;

[0119] S26: Calculate the weight of the WPD beamformer using the estimated value of the noise cosquare matrix and the direction vector, and multiply it by the amplitude spectrum to obtain the amplitude spectrum of the enhanced speech;

[0120] S27 combines the phase spectrum of the noisy speech and the amplitude spectrum of the enhanced speech to obtain the final enhanced speech.

[0121] In an embodiment, two microphones are used to receive speech signals under noise and reverberation conditions, and each microphone signal received is subjected to preliminary speech enhancement through a trained single-channel neural network to remove noise in the same direction as the target speech. The adaptive mask predicted by the neural network is used for subsequent directional vector estimation. Unlike traditional positioning that blindly relies on signal energy, AM provides a view of the target signal, can accurately identify the speech-dominant time-frequency unit (i.e., TF unit) on each microphone channel, and only uses these TF units for sound source positioning, so that higher positioning accuracy can be obtained under noise and reverberation conditions. The directional vector estimated using the mask-weighted covariance matrix can be directly applied to microphone arrays of different geometric shapes and numbers. Finally, the calculated weights are input into the WPD beamformer to remove noisy speech in different directions from the target speech and suppress room reverberation, thereby obtaining enhanced speech with good speech quality and high intelligibility.

[0122] The improved WPD convolution beamformer of this embodiment performs preliminary speech enhancement on each received microphone signal through a single-channel neural network, removes noise in the same direction as the target speech, and then uses the adaptive mask predicted by the neural network for subsequent directional vector estimation. The mask can identify the speech-dominated TF units on each microphone channel and only use these TF units for sound source localization, so that higher positioning accuracy can be obtained under noise and reverberation conditions, and the directional vector estimated using mask weighting can be directly applied to microphone arrays of different geometric shapes and numbers; that is, this embodiment combines single-channel and multi-channel signal processing to improve the problem of poor enhancement effect of traditional hearing aid speech enhancement due to hardware conditions, and the enhanced speech obtained has high intelligibility and good speech quality.

[0123] Based on the same inventive concept, an embodiment of the present invention also provides a speech enhancement device based on dual-channel neural network time-frequency masking. Since the principle of the problem solved by the speech enhancement device based on dual-channel neural network time-frequency masking in this embodiment is similar to the speech enhancement method based on dual-channel neural network time-frequency masking in the aforementioned embodiment, the implementation of a speech enhancement device based on dual-channel neural network time-frequency masking can refer to the aforementioned embodiment of a speech enhancement method based on dual-channel neural network time-frequency masking, and the repeated parts will not be repeated.

[0124] In one embodiment, a speech enhancement device based on dual-channel neural network time-frequency masking is provided, the device comprising:

[0125] Preliminary enhancement module: used to train the neural network to estimate the adaptive ratio mask, perform monaural speech enhancement on each microphone signal, and remove background noise in the same direction as the target speech;

[0126] Target positioning module: used to identify the target speech time-frequency unit and the noise time-frequency unit through the adaptive ratio mask, calculate the mask weight representing the importance of the time-frequency unit to DOA estimation; combine the spatial covariance matrix, calculate the speech covariance matrix from the mask weight of the target speech time-frequency unit, calculate the noise covariance matrix from the mask weight of the noise time-frequency unit, and estimate the direction vector required by the beamformer through the speech covariance matrix to locate the target speech;

[0127] Speech enhancement module: It is used to calculate the weight of the WPD beamformer by combining the noise covariance matrix and the direction vector estimated by the speech covariance matrix. The beamformer removes the noise in different directions from the target speech and suppresses the reverberation to obtain the final enhanced speech.

[0128] Based on the same inventive concept, the embodiments of the present invention also provide a hearing aid device and a storage medium. Since the principles of the problems solved by the embodiments of the hearing aid device and the storage medium are similar to those of the aforementioned embodiment of a speech enhancement method based on dual-channel neural network time-frequency masking, the implementation of the hearing aid device and the storage medium can refer to the aforementioned embodiment of a speech enhancement method based on dual-channel neural network time-frequency masking, and the repeated parts will not be repeated.

[0129] As another aspect of an embodiment of the present invention, a hearing aid device is provided, comprising: a processor and a memory, wherein the memory is used to store computer program code, and the computer program code includes computer instructions. When the processor executes the computer instructions, the hearing aid device executes any of the above-mentioned speech enhancement methods based on dual-channel neural network time-frequency masking.

[0130] As another aspect of an embodiment of the present invention, a storage medium is provided, in which a computer program is stored. The computer program includes program instructions, and when the program instructions are executed by a processor of a terminal device, the processor executes any of the above-mentioned speech enhancement methods based on dual-channel neural network time-frequency masking.

[0131] The method of this embodiment is evaluated using a dual microphone device to perform speech enhancement in a reverberant environment with diffuse noise. Figure 6As shown in the figure, RIRs are generated by using a room impulse response (RIR) generator based on an image method. A target speech speaker and an interference speaker are set in the range of -90 degrees to 90 degrees with an interval of 45 degrees, and the positions are randomly placed in any two of the five directions. The distance between each speaker and the center of the array is 1.5m, the room size is 8x8x3m, and two microphones are placed in the center of the room. The distance between the two microphones is 0.2 meters, and the height is set to 1.5 meters. The reverberation time (Reverberation time, T60) of each mixture is randomly selected from 0.0 to 1.0 with an interval of 0.2 seconds. The IEEE speech library and NOISE-92 noise library are used for experiments. The neural network is trained using 50 pure speech sounds to create 450 training sets with signal-to-noise ratios of -5dB, 0dB and 5dB under White, Babble and F16 noise. In the algorithm testing phase, 10 groups of speech sounds are selected to create 90 test sets under the same conditions.

[0132] See also Figure 7 , Figure 8 ,Comparing the two figures, it can be seen that IRM suppresses noise while introducing some new noise and cannot maintain the speech structure well. The speech spectrum structure of AM remains intact and the residual noise is small. It is verified that the mask AM in the embodiment of the present invention can not only effectively identify the time-frequency unit dominated by speech or noise, but also ensure the speech quality.

[0133] The effectiveness and advantages of the speech enhancement algorithm based on dual-channel neural network time-frequency masking in the embodiment of the present invention are verified through comparative analysis of the following three groups of algorithms.

[0134] Algorithm 1: Use the WPD beamforming algorithm to directly perform dual-channel speech enhancement.

[0135] Algorithm 2: Speech enhancement algorithm using dual-channel neural network and AM-weighted GCC-PHAT.

[0136] Algorithm 3: Use the speech enhancement algorithm based on dual-channel neural network time-frequency masking proposed in this paper.

[0137] Among them, the traditional WPD-based convolutional beamformer hearing aid speech enhancement method calculates the arrival time difference through methods such as GCC-PHAT, and then estimates the direction vector in combination with the geometric assumptions of the microphone array. Assuming there is only one target speech, the physical model of the signal received by the dual microphone array in a noisy and reverberant environment is:

[0138] y(t,f)=v(f)s(t,f)+h(t,f)+n(t,f), where s(t,f) represents the short-time Fourier transform of the target speech at time t and frequency f, v(f) represents the direction vector of the microphone array, v(f)s(t,f) and h(t,f) represent the direct speech signal and the early and late echoes of the target speech, and y(t,f) and n(t,f) represent the mixed signal received by the microphone and the noise signal with echo. The specific method is as follows:

[0139] Compute the power-normalized spatial covariance matrix R with prediction delays.

[0140]

[0141] Where σ t 2 represents the time-varying power of the signal, and y(t,f) represents the microphone signal.

[0142] Compute the weights ω of the WPD beamformer.

[0143]

[0144] Convolutional beamformer optimized based on unified WPD criterion.

[0145] ω H v=1

[0146] The resulting enhanced signal for

[0147]

[0148] Among them, the comparison of SegSNR under the three algorithms is shown in Table 1, the comparison of STOI under the three algorithms is shown in Table 2, and the comparison of PESQ under the three algorithms is shown in Table 3.

[0149] Table 1 SegSNR comparison

[0150]

[0151] Table 2 STOI comparison

[0152]

[0153] Table 3 PESQ comparison

[0154]

[0155]

[0156] Fig. 9The figure shows the average comparison of the enhancement effects of the three algorithms under different background noises. In order to intuitively show the effectiveness of the proposed algorithm, the speech with a reverberation time of 0.3s, a sampling frequency of 16000HZ, a frame length of 256, a frame overlap of 64, and Babble noise pollution is taken as an example, and the enhanced speech under three experimental conditions is given respectively. The time domain waveform of the comparison of the enhancement effects of the three algorithms under the reverberation time of 0.3s and Babble noise is shown in Figure 1. Fig.10 As shown; the reverberation time is 0.3 seconds, and the spectrograms of the three algorithms under Babble noise enhancement effects are as follows Fig.11 shown.

[0157] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer-usable program code.

[0158] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (apparatus), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0159] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0160] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0161] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A speech enhancement method based on dual-channel neural network time-frequency masking, characterized in that: The method comprises: Train a neural network to estimate an adaptive ratio mask to perform monaural speech enhancement on each microphone signal and remove background noise in the same direction as the target speech. Identifying the target speech time-frequency unit and the noise time-frequency unit by the adaptive ratio mask, and calculating the mask weight representing the importance of the time-frequency unit to the DOA estimation; Combined with the spatial covariance matrix, the speech covariance matrix is ​​calculated by the mask weight of the target speech time-frequency unit, and the noise covariance matrix is ​​calculated by the mask weight of the noise time-frequency unit. The direction vector of the WPD beamformer is calculated by the speech covariance matrix to locate the target speech. Combine the direction vector estimated by the noise covariance matrix and the speech covariance matrix to calculate the weight of the WPD beamformer, remove the noise in different directions from the target speech and suppress the reverberation, and obtain the final enhanced speech; Wherein, the method of "training a neural network to estimate an adaptive ratio mask" comprises: A scaling factor is constructed using the signal-to-noise ratio, and an adaptive ratio mask is established which adaptively adjusts the ratio between the ideal binary mask and the ideal ratio mask according to the signal-to-noise ratio of the time-frequency unit; Taking the direct speech signal as the target and the other components as noise, the mask value of each microphone signal is calculated separately; Wherein, the adaptive ratio mask is: AM i (t,f) = (1 - α(t,f))·IBM i (t,f) + α(t,f)·IRM i (t,f) where: IBM means ideal binary mask, IRM means ideal ratio mask; in: Ed(t,f) represents the energy of the direct signal of the i-th microphone, E h (t,f)+E n (t,f) represents the energy of the i-th microphone noise; SNR(t,f) represents the signal-to-noise ratio of the time-frequency unit, S 2 (t,f),N 2 (t, f) represent the energy of speech and noise respectively; λ and β are used to adjust the shape of the exponential function, and their values ​​are set to -5 and 2 through experimental verification.

2. The speech enhancement method based on dual-channel neural network time-frequency masking as claimed in claim 1, characterized in that: The training method of the "neural network" includes: With the goal of minimizing the mean square error function, the multi-resolution auditory cepstral coefficient features and adaptive ratio mask of the speech in the training set are extracted as the input of the model. Both the input and output are normalized by mean variance, and the neural network model is trained and saved. The multi-resolution auditory cepstral coefficient feature includes four cochlear representations of different resolutions, and uses a nonlinear power function that conforms to human auditory perception to compress signal energy and discrete cosine transform decorrelation.

3. The speech enhancement method based on dual-channel neural network time-frequency masking as claimed in claim 2, characterized in that: The neural network model includes a 4-layer structure, including two hidden layers, each hidden layer is provided with 1024 nodes, and the output layer is provided with 64 nodes. The hidden layer uses a linear rectification activation function, and the output layer uses a Sigmoid activation function.

4. The speech enhancement method based on dual-channel neural network time-frequency masking as claimed in claim 3, characterized in that: The method further comprises: Using the adaptive ratio mask of each microphone signal, the mask weights η(t,f) and ξ(t,f) representing the importance of each time-frequency unit to DOA estimation are calculated; in: i represents the i-th microphone signal, D represents the total number of microphone signals, D=2; Combining the adaptive ratio mask and the power-normalized spatial covariance matrix R with prediction delay, the ratio mask weighted noise covariance matrix τ and speech covariance matrix α are obtained; Estimating direction vector from speech covariance matrix Where P{g} represents the calculated principal eigenvector; Given the time-varying power and direction vector required by the signal, calculate the weight ω of the WPD beamformer, Get enhanced signal Among them, y(t,f) represents the mixed signal received by the microphone.

5. The speech enhancement method based on dual-channel neural network time-frequency masking as claimed in claim 4, characterized in that: The method further comprises: Using dual microphones to receive a speech signal containing noise and reverberation, and performing a short-time Fourier transform on the speech signal to obtain a noisy speech spectrum, wherein the noisy speech spectrum includes an amplitude spectrum and a phase spectrum; The amplitude spectrum of each microphone signal is input into a single-channel neural network to estimate the adaptive ratio mask and preliminary speech enhancement; Multiply the adaptive ratio masks of each microphone signal and calculate the mask weight representing the importance of each time-frequency unit to the DOA estimation; Combining the mask weights and a power-normalized speech signal spatial covariance matrix with prediction delay to obtain a ratio-masked weighted noise covariance matrix and a speech covariance matrix; Calculate an estimate of the direction vector using the speech covariance matrix; Calculating the weight of the WPD beamformer using the estimated values ​​of the noise covariance matrix and the direction vector, and multiplying the weight with the amplitude spectrum to obtain the amplitude spectrum of the enhanced speech; The final enhanced speech is obtained by combining the phase spectrum of the noisy speech and the amplitude spectrum of the enhanced speech.

6. A speech enhancement device based on dual-channel neural network time-frequency masking, characterized in that: The device is based on the speech enhancement method according to any one of claims 1 to 5, and comprises: Preliminary enhancement module: used to train the neural network to estimate the adaptive ratio mask, perform monaural speech enhancement on each microphone signal, and remove background noise in the same direction as the target speech; Target localization module: used to identify the target speech time-frequency unit and the noise time-frequency unit through the adaptive ratio mask, and calculate the mask weight representing the importance of the time-frequency unit to the DOA estimation; Combined with the spatial covariance matrix, the speech covariance matrix is ​​calculated by the mask weight of the target speech time-frequency unit, the noise covariance matrix is ​​calculated by the mask weight of the noise time-frequency unit, and the direction vector required by the beamformer is estimated through the speech covariance matrix to locate the target speech; The speech enhancement module calculates the weight of the WPD beamformer by combining the noise covariance matrix and the direction vector estimated by the speech covariance matrix. The beamformer removes noise in different directions from the target speech and suppresses reverberation to obtain the final enhanced speech.

7. A hearing aid device, characterized in that: include: A processor and a memory, wherein the memory is used to store computer program codes, wherein the computer program codes include computer instructions, and when the processor executes the computer instructions, the hearing aid device executes the speech enhancement method based on dual-channel neural network time-frequency masking as described in any one of claims 1 to 5.

8. A storage medium, characterized in that: The storage medium stores a computer program, which includes program instructions. When the program instructions are executed by a processor of a terminal device, the processor executes the speech enhancement method based on dual-channel neural network time-frequency masking as described in any one of claims 1 to 5.

Images