A smart hearing aid method and device based on spectrum enhancement

Abstract

This invention provides a method and device for intelligent hearing aids based on spectrum enhancement, relating to the field of speech signal processing technology. The method includes the following steps: acquiring audio signals and preprocessing them; converting the audio signals into a spectrogram, labeling the data, and dividing it into training, validation, and test sets; creating a spectrum enhancement model and training it using the spectrogram of the training set; iterating the spectrum enhancement model multiple times, tuning hyperparameters until the model converges, and evaluating the model; and inputting real-time audio signals for spectrum enhancement processing and outputting the results. This invention enhances the input audio signal by constructing and training a spectrum enhancement model, which can adaptively adjust the spectral gain of the speech signal according to different noise types and intensities in different frequency bands, effectively improving the quality of the speech signal and enhancing the hearing aid effect.

Description

Technical Field

[0001] This invention relates to the field of speech signal processing technology, and in particular to an intelligent hearing aid method and device based on spectrum enhancement. Background Technology

[0002] In recent years, with increasing societal attention to hearing impairment, intelligent hearing aid technology has developed rapidly, becoming an important means of improving the quality of life for people with hearing impairments. Traditional hearing aids improve the audibility of speech signals by amplifying them; however, due to environmental noise interference, the effectiveness of traditional hearing aids often fails to meet user needs in complex noisy environments. Therefore, modern intelligent hearing aid technology has gradually incorporated methods based on signal processing and deep learning. Through intelligent algorithms, it optimizes audio signals, improving the clarity and intelligibility of speech signals, thereby enhancing the auditory experience for people with hearing impairments.

[0003] Spectrum enhancement methods are gradually becoming an important means of improving hearing aid technology due to their ability to process speech signals more accurately in the frequency domain. Through spectrum enhancement, the system can suppress background noise in noisy environments, enhance the audibility of speech signals, improve speech clarity, and thus significantly improve the user's auditory experience.

[0004] While existing deep learning-based intelligent hearing aids have improved hearing aid effectiveness to some extent, they still have some limitations. Existing methods are poorly adaptable to different types of noise, especially in environments with significant variations in noise type and intensity, where the spectral enhancement effect is not ideal.

[0005] How to solve the above-mentioned technical problems is the challenge facing this invention. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides an intelligent hearing aid method and device that adaptively adjusts the spectral gain based on different noise types and intensities to effectively improve hearing aid performance.

[0007] The technical solution adopted by this invention to solve its technical problem is as follows: This invention provides a smart hearing aid method based on spectrum enhancement, comprising the following steps:

[0008] Acquire audio signals and preprocess them;

[0009] The audio signal is converted into a spectrogram, the data is labeled, and the training set, validation set, and test set are divided.

[0010] The training set, validation set, and test set comprise 70%, 20%, and 10% of the collected audio signals, respectively. The training set is used to train the model. The model learns the relationships between sample features and labels in the training set to adjust its parameters, enabling it to fit patterns in the training data and thus predict new data. The validation set is used to evaluate the model's performance during training and to adjust its hyperparameters (such as learning rate, number of network layers, and number of neurons). The test set is used to ultimately evaluate the model's generalization ability, i.e., its ability to adapt to new data different from the training data.

[0011] Create a spectrum enhancement model and input the spectrum of the training set for model training;

[0012] The spectrum enhancement model was iterated multiple times, and hyperparameters were tuned until the model converged and then evaluated.

[0013] The input real-time audio signal is subjected to spectrum enhancement processing and then output.

[0014] The spectrum enhancement model employs a CNN convolutional neural network model.

[0015] CNNs excel at processing two-dimensional data (such as images and spectrograms) and are widely used in speech enhancement and audio signal processing tasks. The input signal in this invention is a spectrogram, rather than the original continuously sampled signal. By transforming the audio signal from the time domain to a frequency domain spectrogram using Short-Time Fourier Transform (STFT), the frequency characteristics of the speech signal can be better preserved. The spectrogram is a two-dimensional structured data set, and CNNs can extract local time- and frequency-related features through convolution operations.

[0016] The specific steps for creating a spectrum enhancement model and training the model using the spectrum graph of the training set are as follows:

[0017] After training the audio signal input to the spectrum enhancement model, forward propagation is performed to extract the type and intensity distribution of noise in the spectrum of the training audio signal, calculate the corresponding speech signal gain, and output the spectrum after gain.

[0018] It should be noted that during the forward propagation process, the model extracts the type and intensity distribution of noise in the training audio signal in the spectrogram. For example, convolutional layers can slide the convolution kernel on the spectrogram to extract features at different locations and frequency ranges. These features can reflect both local and global information about the noise. Pooling layers can reduce the dimensionality of the features, extract the main feature information, and reduce computational cost and the risk of model overfitting.

[0019] The formula for calculating speech signal gain is as follows:

[0020] ,

[0021] In the formula, For speech signal gain, For time, For frequency, for Time, frequency The amplitude of the speech signal at that location, for Time, frequency The amplitude of the noise signal at that location, for Time, frequency Signal-to-noise ratio at the location. The correlation constant is adjusted for the signal-to-noise ratio. for Time, frequency The dynamic adjustment factor at the location, For user-defined functions, This is a modulation function based on the signal-to-noise ratio.

[0022] According to different frequencies The amplitude of the voice signal at the location noise signal amplitude and signal-to-noise ratio Combined with dynamic adjustment factors Calculate the appropriate gain to enhance the spectrum of the speech signal, thereby improving the quality of the speech signal and the auditory experience.

[0023] ,

[0024] In the formula, As noise intensity weight, The weights are for the noise type.

[0025] Combined with user-defined functions This allows the dynamic adjustment factor to consider not only noise intensity and type, but also the relative amplitudes of the speech and noise signals, thus more comprehensively reflecting the gain adjustment requirements under different conditions. When the noise signal amplitude... When it is large, Closer to 1, making Larger gain allows for stronger gain adjustments in noisier frequency bands; while noise type weighting... and noise intensity weight Differentiated gain adjustments can be made based on different types of noise (such as mechanical noise, electrical noise, etc.) and different noise intensities.

[0026] It should be noted that, The acquisition method can be based on experimental data statistics, typically determined according to the actual gain requirements in different signal-to-noise ratio (SNR) ranges. For example, the weight is smaller at higher SNRs and larger at lower SNRs. γ represents the importance of different noise types to signal processing; for example, mechanical noise, electrical noise, and background noise have different impacts on speech quality. It can be assigned values ​​through data analysis or expert experience. For example, using manually annotated datasets, speech clarity under different types of noise can be scored, and then the weights can be calculated using normalization.

[0027] In the iteration of the model, and The optimal value for the corresponding noise type and intensity is found using the gradient descent algorithm. Since noise environments and signal conditions are diverse and constantly changing in real-world applications, a dynamic adjustment factor allows the spectral enhancement model to adapt to these changes. The dynamic adjustment factor combines information on noise intensity and type, ensuring that gain adjustment does not rely on a single factor but comprehensively considers the multifaceted characteristics of noise and its relationship with the speech signal. This allows for the suppression of noise while preserving and enhancing the useful components of the speech signal as much as possible, achieving better speech enhancement and auditory experience. For example, for a specific type of noise (composed of...), the optimal value is found for the corresponding noise type and intensity. (As a manifestation), if it has a significant impact on the speech signal, even if the noise intensity is not particularly high (but through...) (The dynamic adjustment factor will also adjust the gain accordingly to better cope with the impact of this noise on speech and achieve a balance between noise suppression and speech enhancement).

[0028] ,

[0029] The value of the function depends on the relative amplitude of the speech signal and the noise signal. By introducing a nonlinear adjustment through a sine function, the function value will increase when the speech signal is relatively strong, thereby enhancing the gain adjustment of the speech signal; when the noise signal is strong, the function value will decrease, reducing the gain adjustment of the speech signal.

[0030] ,

[0031] Based on signal-to-noise ratio The magnitude of the gain adjustment is determined by the value of the signal-to-noise ratio (SNR), and its contribution to the overall gain varies depending on the SNR. Its design aims to enhance gain adjustment when noise is strong (low SNR) and reduce gain adjustment when the speech signal is strong (high SNR), working in conjunction with other components to achieve dynamic gain adjustment based on the SNR.

[0032] The spectrum after gain is expressed by the following formula:

[0033] ,

[0034] In the formula, This is the spectrum after gain, where 'a' is the exponential adjustment constant. X This is the initial spectrum.

[0035] Based on signal-to-noise ratio Through the exponential function Further adjustments were made to the results to prevent the spectrum from being over-adjusted under high noise conditions due to the rapid change of the exponential function, or to prevent unwanted nonlinear behavior from occurring when the noise and signal strengths are comparable.

[0036] when When the volume is relatively large (the voice signal is relatively strong), The value is relatively small. It is also relatively small, an exponential function. The value will be closer to 1, making The value is relatively small, thus affecting The adjustments are relatively small, avoiding excessive amplification of the speech signal when the speech signal itself is already strong.

[0037] when When the noise level is relatively low (and the noise level is relatively high), The value is relatively large. The exponential function will also increase accordingly. The value will be closer to 0, at which point... The value is closer to 1, for The adjustment is significant, thereby enhancing the speech signal in the noisy frequency band, which helps to suppress noise and highlight the speech signal.

[0038] Define the loss function for the spectrum enhancement model and calculate the loss.

[0039] The loss function is defined as follows:

[0040]

[0041] In the formula, For the total loss, These are weighting coefficients used to balance the contributions of different losses to model optimization, and can be adjusted based on experiments; Loss for classifying noise types For noise intensity regression loss, For speech signal gain loss;

[0042] The gradient of the loss function is calculated using the backpropagation algorithm, and the parameters of the spectral enhancement model are updated using the gradient descent algorithm.

[0043] It should be noted that the backpropagation algorithm starts from the output layer and, according to the chain rule, progressively calculates the gradient of the parameters of each layer with respect to the loss function. For example, for the weights and biases of a convolutional layer, the gradient values ​​are obtained by calculating their partial derivatives with respect to the loss function. Then, the gradient descent algorithm is used to update the parameters of the spectral augmentation model. The gradient descent algorithm updates the model parameters according to a certain learning rate based on the calculated gradients.

[0044] Furthermore, the audio signal is a speech signal containing different types and intensities of noise.

[0045] Noise types include mechanical noise, electrical noise, ambient background noise, and echo noise;

[0046] Noise intensity includes weak noise with a signal-to-noise ratio (SNR) higher than +5 dB; medium noise with an SNR between -10 dB and +5 dB; and strong noise with an SNR lower than -10 dB.

[0047] Furthermore, the preprocessing includes silent cropping, normalization, frame splitting, and dynamic windowing.

[0048] By segmenting the signal into frames and calculating the energy of each frame, an energy threshold is set. When the energy is less than the preset threshold, the frame is considered a silent segment. This process of cropping silent segments in the training audio signal reduces the computational load of the model and improves the convergence speed of the model.

[0049] Normalization eliminates amplitude inconsistencies caused by differences in audio source intensity, thereby avoiding training bias caused by volume differences.

[0050] Based on the sampling rate and predetermined frame length and frame shift, the start and end positions of each frame are calculated. Then, the corresponding signal segments are extracted to achieve signal framing. Each frame typically contains tens of milliseconds of audio signal. Audio signals are usually non-stationary, and framing allows for analysis within a shorter time window, enabling the model to capture more detailed time-frequency features and effectively improve the accuracy of the analysis.

[0051] By multiplying each frame signal with a dynamic window function, a windowed frame signal is obtained, which reduces discontinuities at frame boundaries, reduces spectral leakage during signal processing, and improves the spectral representation quality of the signal, thereby making the spectrogram smoother and more accurate, which is beneficial for subsequent spectral enhancement and feature extraction.

[0052] Furthermore, the step of converting the audio signal into a spectrogram includes performing a short-time Fourier transform on the preprocessed audio signal, arranging the spectral data into a spectrogram, and performing a logarithmic transform and scaling on the spectrogram.

[0053] The audio signal is converted from the time domain to the frequency domain using STFT. The spectral data (usually amplitude spectrum) of each frame is stacked in chronological order to obtain a matrix, where the rows of the matrix represent frequencies. The column represents time. Each cell value represents the amplitude information of the speech signal and noise signal at the corresponding time and frequency, ultimately forming a two-dimensional spectrogram.

[0054] The formula for calculating the logarithmic transformation is as follows:

[0055] ,

[0056] In the formula, The amplitude is shown in the original spectrum. This represents the amplitude of the spectrum after logarithmic transformation. Since the amplitude of the spectrum may contain zero values ​​(e.g., silent portions), directly performing a logarithmic transformation on zero values ​​can lead to computational instability. Therefore, a small constant of 1 is added to avoid zero values ​​in the logarithm. Compressing the dynamic range of the spectrum through logarithmic transformation results in a more balanced contrast between high-energy and low-energy components, helping to highlight the characteristics of the low-energy portion and improving noise suppression and speech recognition performance.

[0057] By scaling the spectrogram to a uniform scale, it becomes more suitable for subsequent processing, avoiding poor processing results due to excessive amplitude differences.

[0058] Furthermore, the noise type classification loss is calculated using the cross-entropy loss function, and the noise intensity regression loss and the speech signal gain loss are calculated using the mean square error loss function.

[0059] Noise type classification loss is used for classification tasks that distinguish noise types (such as mechanical noise, electrical noise, etc.), and typically employs the cross-entropy loss function:

[0060] ;

[0061] in, These are labels for real noise types. It is the predicted probability of the noise type output by the model. That is the total number of samples.

[0062] Noise intensity regression loss is used to predict noise intensity, and the mean squared error (MSE) is usually used as the loss function.

[0063] ;

[0064] in, This is the actual noise intensity. is the noise intensity predicted by the model, and M is the total number of samples.

[0065] Speech signal enhancement loss is used to evaluate the difference between the enhanced speech signal and the original signal, and is calculated using the mean square error of the spectral amplitude values ​​before and after enhancement.

[0066] ;

[0067] in, This is the amplitude value before enhancement. This is the enhanced amplitude value. It is the number of frequency components.

[0068] Furthermore, the process of iterating the spectrum enhancement model multiple times, tuning hyperparameters until the model converges, and evaluating the model based on the PESQ score specifically involves:

[0069] The spectrum enhancement model is trained iteratively multiple times on the training set according to the defined model training method.

[0070] By repeatedly performing forward propagation, loss calculation, backpropagation, and parameter update, the model's parameters are gradually optimized, resulting in continuous improvement in the model's performance on the training set. Ultimately, this enables the model to better perform spectral enhancement processing on new audio signals.

[0071] Set a loss threshold and a maximum number of iterations. After each training cycle, calculate the loss on the validation set. If the validation set loss gradually decreases and the change is less than the preset loss threshold in several consecutive training iterations, the model is considered to have converged. If the validation set loss gradually increases or remains unchanged and the change is greater than the preset loss threshold in several consecutive training iterations, the model is considered to have not converged, and the model and hyperparameters are adjusted. If the model has not converged after reaching the maximum number of iterations, the model and hyperparameters are adjusted.

[0072] Setting a maximum number of iterations limits training time and prevents indefinite training, helping to control the consumption of computational resources (such as CPU and GPU usage time and memory usage). It also prevents overfitting caused by overtraining. Furthermore, if the algorithm itself has problems or the data is unsuitable for the model, the algorithm may fail to converge to a stable solution. For example, when using an inappropriate learning rate for gradient descent, the algorithm may oscillate within the parameter space and fail to converge; in this case, the maximum number of iterations can limit this ineffective iterative process.

[0073] The trained model is loaded onto the test set, and its performance is judged by calculating the objective speech quality PESQ score of the speech signal. During the calculation, the start and end points of the original audio signal and the gain-processed audio signal are first synchronized and aligned. The original audio signal and the gain-processed audio signal are then divided into several time frames (typically each frame is 20-30 milliseconds long, with an overlap of 50%-75%). Then, according to the standard PESQ algorithm defined by the International Telecommunication Union (ITU), the auditory perception model based on the human ear conduction model in the PESQ algorithm is loaded. The spectral changes of the signal are mapped to an output similar to human ear perception through a filter bank. Features of each frame are extracted through the auditory perception model, including spectral distortion and temporal variations. Based on the analysis of each frame, a distortion metric similar to human ear perception is calculated by comparing the spectral and temporal features of the original audio signal and the gain-processed audio signal. Finally, the PESQ algorithm integrates the perceived quality of the entire audio signal to generate an overall PESQ score.

[0074] When the PESQ score is greater than or equal to 3.0 and less than 3.5, the model performance is considered satisfactory; when the PESQ score is greater than or equal to 3.5 and less than 4, the model performance is considered good; when the PESQ score is greater than or equal to 4, the model performance is considered excellent; when the PESQ score is less than 3.5, the model performance is considered unsatisfactory, and the model and hyperparameters are adjusted and the iteration is restarted.

[0075] PESQ scores typically range from 4-5: Excellent, near-perfect sound quality. 3-4: Good, clear sound quality with only minor flaws. 2-3: Acceptable, but with noticeable defects. 1-2: Poor, poor sound quality, not clear enough. 0-1: Very poor, incomprehensible sound quality, almost unacceptable. When the PESQ score is 3.0 or higher, the speech signal quality is high and meets the speech signal clarity requirements in most situations.

[0076] Furthermore, the hyperparameter tuning employs a Bayesian optimization algorithm.

[0077] It should be noted that in machine learning models, hyperparameters are usually parameters pre-set during training, such as learning rate and batch size. These parameters are not directly reflected in the data representation, but indirectly determine the model's performance by influencing the model training process and convergence behavior.

[0078] The significance of hyperparameter tuning: The "hyperparameter tuning" mentioned in this invention is for optimizing the model's training process and performance, rather than adjusting specific input or output data. Therefore, although hyperparameters do not have a direct data representation, adjusting their value range can improve model performance and convergence speed, which is a common technique for optimizing machine learning models.

[0079] Furthermore, the input real-time audio signal undergoes spectral enhancement processing and is output as follows:

[0080] Convert real-time audio signals into a spectrogram.

[0081] To achieve the conversion from the time domain to the frequency domain, a short-time Fourier transform (STFT) is used. It divides the audio signal into multiple time segments, performs a Fourier transform on each segment, and combines them to obtain a two-dimensional spectrum. The horizontal axis represents time, the vertical axis represents frequency, and the color represents amplitude.

[0082] Input the spectrogram into the spectrum enhancement model, calculate the speech signal gain based on the noise type and noise intensity, and output the enhanced spectrogram.

[0083] After the spectrogram is input into the spectrum enhancement model, the convolutional neural network analyzes the input spectrogram, identifies the noise type and noise intensity of different frequency bands, calculates the gain of the speech signal in different frequency bands according to the noise type and noise intensity, calculates the gain of the speech signal in different frequency bands through the defined gain function, multiplies the calculated gain with the spectrogram, performs gain processing on the spectrogram, and finally outputs the enhanced spectrogram.

[0084] Perform an inverse short-time Fourier transform on the gained spectrum to output the audio signal.

[0085] The gain-processed spectrogram is then transformed back to the time domain using an inverse short-time Fourier transform (ISTFT). The purpose of this process is to recover the audio signal and make it suitable for playback through hearing aids.

[0086] The present invention also provides a smart hearing aid device based on spectrum enhancement, comprising:

[0087] The signal input module is used to acquire audio signals and perform preprocessing operations on the audio signals;

[0088] The spectrum conversion and data annotation module is used to convert the input audio signal into a spectrum and perform data annotation.

[0089] The model training module is used to build and iteratively fine-tune the spectrum enhancement model;

[0090] The spectrum gain calculation module is used to calculate the gain of the speech signal based on the noise type and noise intensity, and output the spectrum after gain.

[0091] The signal output module is used to convert the gained spectrogram into an audio signal for output.

[0092] The beneficial effects of this invention are as follows: It adaptively adjusts the spectral gain according to different noise types and intensities, effectively improving the hearing aid effect. It prunes silent segments in the training audio signal, reducing model computation and improving convergence speed. Normalization eliminates amplitude inconsistencies caused by differences in audio source intensity, avoiding training bias due to volume differences. Framing and dynamic windowing enable the model to capture more detailed time-frequency features, reducing discontinuities at frame boundaries and spectral leakage during signal processing, thereby improving the spectral representation quality of the signal. Logarithmic transformation reduces the dynamic range of the spectral amplitude, preventing excessive differences between high-intensity and low-intensity signals from overwhelming weaker signals, thus improving the robustness and accuracy of the speech enhancement algorithm. Scaling the spectrogram to a uniform scale avoids learning bias caused by different data magnitudes. By using a convolutional neural network for forward propagation to extract the distribution of noise type and intensity on the spectrogram and calculating the corresponding gain output, adaptive adjustment for different noise types and intensities is achieved. Gradient descent algorithm is used to update model parameters, and Bayesian optimization algorithm is used to tune hyperparameters, further enhancing the convergence speed and accuracy of the model. Attached Figure Description

[0093] Figure 1 This is a diagram illustrating the method steps of the present invention.

[0094] Figure 2 This is a schematic diagram of the device module of the present invention. Detailed Implementation

[0095] To clearly illustrate the technical features of this solution, the following detailed implementation method will be used to explain the solution.

[0096] Example 1:

[0097] See Figure 1 As shown, this embodiment provides a smart hearing aid method based on spectrum enhancement, including the following steps:

[0098] S1. Acquire audio signals and preprocess them.

[0099] Audio signals are speech signals containing different types and intensities of noise.

[0100] It should be noted that noise types include mechanical noise, electrical noise, ambient background noise, and echo noise; noise intensity includes weak noise with a signal-to-noise ratio higher than +5dB; moderate noise with a signal-to-noise ratio between -10dB and +5dB; and strong noise with a signal-to-noise ratio lower than -10dB.

[0101] Preprocessing includes silent cropping, normalization, frame splitting, and dynamic windowing.

[0102] By segmenting the signal into frames and calculating the energy of each frame, an energy threshold is set. When the energy is less than the preset threshold, the frame is considered a silent segment. This process of cropping silent segments in the training audio signal reduces the computational load of the model and improves the convergence speed of the model.

[0103] Normalization eliminates amplitude inconsistencies caused by differences in audio source intensity, thereby avoiding training bias caused by volume differences.

[0104] Based on the sampling rate and predetermined frame length and frame shift, the start and end positions of each frame are calculated. Then, the corresponding signal segments are extracted to achieve signal framing. Each frame typically contains tens of milliseconds of audio signal. Audio signals are usually non-stationary, and framing allows for analysis within a shorter time window, enabling the model to capture more detailed time-frequency features and effectively improve the accuracy of the analysis.

[0105] By multiplying each frame signal with a dynamic window function, a windowed frame signal is obtained, which reduces discontinuities at frame boundaries, reduces spectral leakage during signal processing, and improves the spectral representation quality of the signal, thereby making the spectrogram smoother and more accurate, which is beneficial for subsequent spectral enhancement and feature extraction.

[0106] S2. Convert the audio signal into a spectrogram, label the data, and divide it into training, validation, and test sets.

[0107] Converting an audio signal into a spectrogram involves performing a short-time Fourier transform on the preprocessed audio signal, arranging the spectral data into a spectrogram, and performing a logarithmic transform and scaling on the spectrogram.

[0108] The audio signal is converted from the time domain to the frequency domain using STFT. The spectral data (usually amplitude spectrum) of each frame is stacked in chronological order to obtain a matrix, where the rows of the matrix represent frequencies. The column represents time. Each cell value represents the amplitude information of the speech signal and noise signal at the corresponding time and frequency, ultimately forming a two-dimensional spectrogram.

[0109] It should be noted that the formula for calculating the logarithmic transformation is as follows:

[0110] ,

[0111] In the formula, The amplitude is shown in the original spectrum. This represents the amplitude of the spectrum after logarithmic transformation. Since the amplitude of the spectrum may contain zero values ​​(e.g., silent portions), directly performing a logarithmic transformation on zero values ​​can lead to computational instability. Therefore, a small constant of 1 is added to avoid zero values ​​in the logarithm. Compressing the dynamic range of the spectrum through logarithmic transformation results in a more balanced contrast between high-energy and low-energy components, helping to highlight the characteristics of the low-energy portion and improving noise suppression and speech recognition performance.

[0112] By scaling the spectrogram to a uniform scale, it becomes more suitable for subsequent processing, avoiding poor processing results due to excessive amplitude differences.

[0113] The data annotation includes speech signals, noise signals, noise type, and noise intensity.

[0114] By effectively reducing the dynamic range of the spectral amplitude through logarithmic transformation, the difference between high-intensity and low-intensity signals is not too large, reducing the difficulty of model training and preventing the strong signal from drowning out the weak signal, thereby improving the robustness and accuracy of the speech enhancement algorithm.

[0115] Scaling the spectrogram to a uniform scale avoids learning bias caused by different data volumes.

[0116] The training set, validation set, and test set comprise 70%, 20%, and 10% of the collected audio signals, respectively. The training set is used to train the model. The model learns the relationships between sample features and labels in the training set to adjust its parameters, enabling it to fit patterns in the training data and thus predict new data. The validation set is used to evaluate the model's performance during training and to adjust its hyperparameters (such as learning rate, number of network layers, and number of neurons). The test set is used to ultimately evaluate the model's generalization ability, i.e., its ability to adapt to new data different from the training data.

[0117] S3. Create a spectrum enhancement model and input the spectrum of the training set to train the model.

[0118] The spectrum enhancement model employs a CNN (Convolutional Neural Network) model. CNNs excel at processing two-dimensional data (such as images and spectrograms) and are widely used in speech enhancement and audio signal processing tasks. The input signal in this invention is a spectrogram, rather than the original continuously sampled signal. By transforming the audio signal from the time domain to a frequency domain spectrogram through Short-Time Fourier Transform (STFT), the frequency characteristics of the speech signal can be better preserved. The spectrogram is a two-dimensional structured data set, and CNNs can extract local time- and frequency-related features through convolution operations.

[0119] The specific steps for creating a spectrum enhancement model and training the model using the spectrum of the training audio signal are as follows:

[0120] S30. After training the audio signal input to the spectrum enhancement model, perform forward propagation to extract the type and intensity distribution of noise in the spectrum of the training audio signal, calculate the corresponding speech signal gain, and output the spectrum after gain.

[0121] It should be noted that during the forward propagation process, the model extracts the type and intensity distribution of noise in the training audio signal in the spectrogram. For example, convolutional layers can slide the convolution kernel on the spectrogram to extract features at different locations and frequency ranges. These features can reflect both local and global information about the noise. Pooling layers can reduce the dimensionality of the features, extract the main feature information, and reduce computational cost and the risk of model overfitting.

[0122] The formula for calculating speech signal gain is as follows:

[0123] ,

[0124] In the formula, For speech signal gain, For time, For frequency, for Time, frequency The amplitude of the speech signal at that location, for Time, frequency The amplitude of the noise signal at that location, for Time, frequency Signal-to-noise ratio at the location. The correlation constant is adjusted for the signal-to-noise ratio. for Time, frequency The dynamic adjustment factor at the location, For user-defined functions, This is a modulation function based on the signal-to-noise ratio.

[0125] According to different frequencies The amplitude of the voice signal at the location noise signal amplitude and signal-to-noise ratio Combined with dynamic adjustment factors Calculate the appropriate gain to enhance the spectrum of the speech signal, thereby improving the quality of the speech signal and the auditory experience.

[0126] ,

[0127] In the formula, As noise intensity weight, The weights are for the noise type.

[0128] Combined with user-defined functions This allows the dynamic adjustment factor to consider not only noise intensity and type, but also the relative amplitudes of the speech and noise signals, thus more comprehensively reflecting the gain adjustment requirements under different conditions. When the noise signal amplitude... When it is large, Closer to 1, making Larger gain allows for stronger gain adjustments in noisier frequency bands; while noise type weighting... and noise intensity weight Differentiated gain adjustments can be made based on different types of noise (such as mechanical noise, electrical noise, etc.) and different noise intensities.

[0129] It should be noted that, The acquisition method can be based on experimental data statistics, typically determined according to the actual gain requirements in different signal-to-noise ratio (SNR) ranges. For example, the weight is smaller at higher SNRs and larger at lower SNRs. γ represents the importance of different noise types to signal processing; for example, mechanical noise, electrical noise, and background noise have different impacts on speech quality. It can be assigned values ​​through data analysis or expert experience. For example, using manually annotated datasets, speech clarity under different types of noise can be scored, and then the weights can be calculated using normalization.

[0130] Furthermore, in the iteration of the model, and The optimal value for the corresponding noise type and intensity is found using the gradient descent algorithm. Since noise environments and signal conditions are diverse and constantly changing in real-world applications, a dynamic adjustment factor allows the spectral enhancement model to adapt to these changes. The dynamic adjustment factor combines information on noise intensity and type, ensuring that gain adjustment does not rely on a single factor but comprehensively considers the multifaceted characteristics of noise and its relationship with the speech signal. This allows for the suppression of noise while preserving and enhancing the useful components of the speech signal as much as possible, achieving better speech enhancement and auditory experience. For example, for a specific type of noise (composed of...), the optimal value is found for the corresponding noise type and intensity. (As a manifestation), if it has a significant impact on the speech signal, even if the noise intensity is not particularly high (but through...) (The dynamic adjustment factor will also adjust the gain accordingly to better cope with the impact of this noise on speech and achieve a balance between noise suppression and speech enhancement).

[0131] ,

[0132] The value of the function depends on the relative amplitude of the speech signal and the noise signal. By introducing a nonlinear adjustment through a sine function, the function value will increase when the speech signal is relatively strong, thereby enhancing the gain adjustment of the speech signal; when the noise signal is strong, the function value will decrease, reducing the gain adjustment of the speech signal.

[0133] ,

[0134] Based on signal-to-noise ratio The magnitude of the gain adjustment is determined by the value of the signal-to-noise ratio (SNR), and its contribution to the overall gain varies depending on the SNR. Its design aims to enhance gain adjustment when noise is strong (low SNR) and reduce gain adjustment when the speech signal is strong (high SNR), working in conjunction with other components to achieve dynamic gain adjustment based on the SNR.

[0135] The spectrum after gain is expressed by the following formula:

[0136] ,

[0137] In the formula, This is the spectrum after gain, where 'a' is the exponential adjustment constant. X This is the initial spectrum.

[0138] Based on signal-to-noise ratio Through the exponential function Further adjustments were made to the results to prevent the spectrum from being over-adjusted under high noise conditions due to the rapid change of the exponential function, or to prevent unwanted nonlinear behavior from occurring when the noise and signal strengths are comparable.

[0139] when When the volume is relatively large (the voice signal is relatively strong), The value is relatively small. It is also relatively small, an exponential function. The value will be closer to 1, making The value is relatively small, thus affecting The adjustments are relatively small, avoiding excessive amplification of the speech signal when the speech signal itself is already strong.

[0140] when When the noise level is relatively low (and the noise level is relatively high), The value is relatively large. The exponential function will also increase accordingly. The value will be closer to 0, at which point... The value is closer to 1, for The adjustment is significant, thereby enhancing the speech signal in the noisy frequency band, which helps to suppress noise and highlight the speech signal.

[0141] S31. Define the loss function for the spectrum enhancement model and calculate the loss.

[0142] The loss function is defined as:

[0143] ;

[0144] In the formula, For the total loss, The weighting coefficients are used to balance the contribution of different losses to model optimization and can be adjusted based on experiments; For noise type classification tasks, the cross-entropy loss function is typically used to distinguish noise types (such as mechanical noise, electrical noise, etc.).

[0145] ;

[0146] in, These are labels for real noise types. It is the predicted probability of the noise type output by the model. That is the total number of samples.

[0147] For noise intensity regression loss used to predict noise intensity, the mean squared error (MSE) is typically used as the loss function.

[0148] ;

[0149] in, This is the actual noise intensity. is the noise intensity predicted by the model, and M is the total number of samples.

[0150] The speech signal enhancement loss is used to evaluate the difference between the enhanced speech signal and the original signal, and is calculated using the mean square error of the spectral amplitude values ​​before and after enhancement.

[0151] ;

[0152] in, This is the amplitude value before enhancement. This is the enhanced amplitude value. It is the number of frequency components.

[0153] S32. Calculate the gradient of the loss function using the backpropagation algorithm, and update the parameters of the spectrum enhancement model using the gradient descent algorithm.

[0154] It should be noted that the backpropagation algorithm starts from the output layer and, according to the chain rule, progressively calculates the gradient of the parameters of each layer with respect to the loss function. For example, for the weights and biases of a convolutional layer, the gradient values ​​are obtained by calculating their partial derivatives with respect to the loss function. Then, the gradient descent algorithm is used to update the parameters of the spectral augmentation model. The gradient descent algorithm updates the model parameters according to a certain learning rate based on the calculated gradients.

[0155] S4. Perform multiple iterations on the spectrum enhancement model, optimize the hyperparameters until the model converges, and evaluate the model based on the PESQ score.

[0156] The spectrum enhancement model is trained iteratively multiple times on the training set according to the defined model training method.

[0157] By repeatedly performing forward propagation, loss calculation, backpropagation, and parameter update, the model's parameters are gradually optimized, resulting in continuous improvement in the model's performance on the training set. Ultimately, this enables the model to better perform spectral enhancement processing on new audio signals.

[0158] Set a loss threshold and a maximum number of iterations. After each training cycle, calculate the loss on the validation set. If the validation set loss gradually decreases and the change is less than the preset loss threshold in several consecutive training iterations, the model is considered to have converged. If the validation set loss gradually increases or remains unchanged and the change is greater than the preset loss threshold in several consecutive training iterations, the model is considered to have not converged, and the model and hyperparameters are adjusted. If the model has not converged after reaching the maximum number of iterations, the model and hyperparameters are adjusted.

[0159] Setting a maximum number of iterations limits training time and prevents indefinite training, helping to control the consumption of computational resources (such as CPU and GPU usage time and memory consumption). It also prevents overfitting caused by overtraining. Furthermore, if the algorithm itself has problems or the data is unsuitable for the model, the algorithm may fail to converge to a stable solution. For example, when using an inappropriate learning rate for gradient descent, the algorithm may oscillate within the parameter space and fail to converge; in this case, the maximum number of iterations can limit this ineffective iterative process.

[0160] The trained model is loaded onto the test set, and its performance is judged by calculating the objective speech quality PESQ score of the speech signal. During the calculation, the start and end points of the original audio signal and the gain-processed audio signal are first synchronized and aligned. The original audio signal and the gain-processed audio signal are then divided into several time frames (typically each frame is 20-30 milliseconds long, with an overlap of 50%-75%). Then, the auditory perception model based on the human ear conduction model of the PESQ algorithm is loaded. The spectral changes of the signal are mapped to an output similar to human ear perception through a filter bank. Features of each frame are extracted through the auditory perception model, including spectral distortion and temporal changes. Based on the analysis of each frame, a distortion metric similar to human ear perception is calculated by comparing the spectral and temporal features of the original audio signal and the gain-processed audio signal. Finally, the PESQ algorithm integrates the perceived quality of the entire audio signal to generate an overall PESQ score.

[0161] When the PESQ score is greater than or equal to 3.0 and less than 3.5, the model performance is considered satisfactory; when the PESQ score is greater than or equal to 3.5 and less than 4, the model performance is considered good; when the PESQ score is greater than or equal to 4, the model performance is considered excellent; when the PESQ score is less than 3.5, the model performance is considered unsatisfactory, and the model and hyperparameters are adjusted and the iteration is restarted.

[0162] PESQ scores typically range from 4-5: Excellent, near-perfect sound quality. 3-4: Good, clear sound quality with only minor flaws. 2-3: Acceptable, but with noticeable defects. 1-2: Poor, poor sound quality, not clear enough. 0-1: Very poor, incomprehensible sound quality, almost unacceptable. When the PESQ score is 3.0 or higher, the speech signal quality is high and meets the speech signal clarity requirements in most situations.

[0163] The hyperparameter tuning employed a Bayesian optimization algorithm.

[0164] It should be noted that in machine learning models, hyperparameters are usually parameters pre-set during training, such as learning rate and batch size. These parameters are not directly reflected in the data representation, but indirectly determine the model's performance by influencing the model training process and convergence behavior.

[0165] The significance of hyperparameter tuning: The "hyperparameter tuning" mentioned in this invention is for optimizing the model's training process and performance, rather than adjusting specific input or output data. Therefore, although hyperparameters do not have a direct data representation, adjusting their value range can improve model performance and convergence speed, which is a common technique for optimizing machine learning models.

[0166] S5: Input real-time audio signal, perform spectrum enhancement processing, and output.

[0167] S50: Convert the real-time audio signal into a spectrogram.

[0168] To achieve the conversion from the time domain to the frequency domain, a short-time Fourier transform (STFT) is used. It divides the audio signal into multiple time segments, performs a Fourier transform on each segment, and combines them to obtain a two-dimensional spectrum. The horizontal axis represents time, the vertical axis represents frequency, and the color represents amplitude.

[0169] S51. Input the spectrum diagram into the spectrum enhancement model, calculate the speech signal gain based on the noise type and noise intensity, and output the enhanced spectrum diagram.

[0170] After the spectrogram is input into the spectrum enhancement model, the convolutional neural network analyzes the input spectrogram, identifies the noise type and noise intensity of different frequency bands, calculates the gain of the speech signal in different frequency bands according to the noise type and noise intensity, calculates the gain of the speech signal in different frequency bands through the defined gain function, multiplies the calculated gain with the spectrogram, performs gain processing on the spectrogram, and finally outputs the enhanced spectrogram.

[0171] S52. Perform an inverse short-time Fourier transform on the gained spectrum to output the audio signal.

[0172] The gain-processed spectrogram is then transformed back to the time domain using an inverse short-time Fourier transform (ISTFT). The purpose of this process is to recover the audio signal and make it suitable for playback through hearing aids.

[0173] Example 2:

[0174] See Figure 2 As shown, this embodiment provides a smart hearing aid device based on spectrum enhancement, including a signal input module, a spectrum conversion and data annotation module, a model training module, a spectrum gain calculation module, and a signal output module;

[0175] The signal input module is used to acquire audio signals and perform preprocessing operations on the audio signals;

[0176] The spectrum conversion and data annotation module is used to convert the input audio signal into a spectrum and perform data annotation.

[0177] The model training module is used to build and iteratively fine-tune the spectrum enhancement module.

[0178] The spectrum gain calculation module is used to calculate the gain of the speech signal based on the noise type and noise intensity, and output the spectrum after gain.

[0179] The signal output module is used to convert the gained spectrogram into an audio signal for output.

[0180] Example 3:

[0181] This embodiment provides a smart hearing aid method based on spectrum enhancement. It scientifically validates the method's effectiveness in improving hearing by calculating the PESQ scores of speech signals before and after spectrum enhancement for different noise types and intensities. The method includes the following steps:

[0182] S1. Acquire audio signals and preprocess them.

[0183] S2. Convert the audio signal into a spectrogram, label the data, and divide it into training, validation, and test sets.

[0184] S3. Create a spectrum enhancement model and input the spectrum of the training set to train the model.

[0185] S4. Perform multiple iterations on the spectrum enhancement model, optimize hyperparameters until the model converges, and evaluate the model.

[0186] S5: Input real-time audio signal, perform spectrum enhancement processing, and output.

[0187] S50: Convert the real-time audio signal into a spectrogram.

[0188] To achieve the conversion from the time domain to the frequency domain, a short-time Fourier transform (STFT) is used. It divides the audio signal into multiple time segments, performs a Fourier transform on each segment, and combines them to obtain a two-dimensional spectrum. The horizontal axis represents time, the vertical axis represents frequency, and the color represents amplitude.

[0189] S51. Input the spectrum diagram into the spectrum enhancement model, calculate the speech signal gain based on the noise type and noise intensity, and output the enhanced spectrum diagram.

[0190] After the spectrogram is input into the spectrum enhancement model, the convolutional neural network analyzes the input spectrogram, identifies the noise type and noise intensity of different frequency bands, calculates the gain of the speech signal in different frequency bands according to the noise type and noise intensity, calculates the gain of the speech signal in different frequency bands through the defined gain function, multiplies the calculated gain with the spectrogram, performs gain processing on the spectrogram, and finally outputs the enhanced spectrogram.

[0191] S52. Perform an inverse short-time Fourier transform on the gained spectrum to output the audio signal.

[0192] The gain-processed spectrogram is then transformed back to the time domain using an inverse short-time Fourier transform (ISTFT). The purpose of this process is to recover the audio signal and make it suitable for playback through hearing aids.

[0193] The effectiveness of this method is illustrated below using audio signals containing mechanical noise and environmental background noise of varying intensities as examples. Specific data is shown in Tables 1 and 2.

[0194]

[0195] Table 1. Speech signal quality after spectral enhancement for different intensities of mechanical noise.

[0196]

[0197] Table 2. Speech signal quality after spectral enhancement for different intensities of ambient background noise.

[0198] As shown in Tables 1 and 2, after the audio signal input is trained into the spectrum enhancement model, the PESQ score of the speech signal is significantly improved compared to the PESQ score of the speech signal before gain processing. The PESQ scores of speech signals containing different noise types and intensities are all improved to above 3.5, which effectively improves the quality of noisy speech signals, enabling them to meet the requirements of speech signal quality in most scenarios and significantly improves the hearing aid effect in complex noisy environments.

[0199] The technical features of this invention not described can be implemented by or using existing technology, and will not be repeated here. Of course, the above description is not a limitation of this invention, and this invention is not limited to the examples above. Any changes, modifications, additions or substitutions made by those skilled in the art within the scope of this invention should also be within the protection scope of this invention.

Claims

1. A smart hearing aid method based on spectrum enhancement, characterized in that, Includes the following steps: Acquire audio signals and preprocess them; The audio signal is converted into a spectrogram, the data is labeled, and the training set, validation set, and test set are divided. Create a spectrum enhancement model and input the spectrum of the training set for model training; The spectrum enhancement model was iterated multiple times, and hyperparameters were tuned until the model converged. The model was then evaluated based on the PESQ score. Input a real-time audio signal, perform spectrum enhancement processing, and output the signal. The spectrum enhancement model employs a CNN convolutional neural network model. The specific steps for creating a spectrum enhancement model and training the model using the spectrum graph of the training set are as follows: After training the audio signal input to the spectrum enhancement model, forward propagation is performed to extract the type and intensity distribution of noise in the spectrum of the training audio signal, calculate the corresponding speech signal gain, and output the spectrum after gain. The formula for calculating speech signal gain is as follows: ， In the formula, For speech signal gain, For time, For frequency, for Time, frequency The amplitude of the speech signal at that location, for Time, frequency The amplitude of the noise signal at that location, For signal-to-noise ratio, The correlation constant is adjusted for the signal-to-noise ratio. for Time, frequency The dynamic adjustment factor at the location, For user-defined functions, Let be the adjustment function based on the signal-to-noise ratio; where, ， In the formula, As noise intensity weight, Weights for noise type; ， ， The spectrum after gain is expressed by the following formula: , In the formula, This is the spectrum after gain, where 'a' is the exponential adjustment constant. X This is the initial spectrum. Define the loss function for the spectrum enhancement model and calculate the loss; The loss function is defined as follows: ， In the formula, For the total loss, These are the weighting coefficients. Loss for classifying noise types For noise intensity regression loss, For speech signal gain loss; The gradient of the loss function is calculated using the backpropagation algorithm, and the parameters of the spectral enhancement model are updated using the gradient descent algorithm.

2. The intelligent hearing aid method based on spectrum enhancement according to claim 1, characterized in that, The audio signal is a speech signal containing different types and intensities of noise; Noise types include mechanical noise, electrical noise, ambient background noise, and echo noise; Noise intensity includes weak noise with a signal-to-noise ratio (SNR) higher than +5 dB; medium noise with an SNR between -10 dB and +5 dB; and strong noise with an SNR lower than -10 dB.

3. The intelligent hearing aid method based on spectrum enhancement according to claim 2, characterized in that, Preprocessing operations include silent cropping, normalization, frame splitting, and dynamic windowing.

4. The intelligent hearing aid method based on spectrum enhancement according to claim 3, characterized in that, The process of converting the audio signal into a spectrogram includes performing a short-time Fourier transform on the preprocessed training audio signal, arranging the spectral data into a spectrogram, and performing a logarithmic transform and scaling on the spectrogram. The data annotation includes annotating the speech signal, noise signal, noise type, and noise intensity.

5. The intelligent hearing aid method based on spectrum enhancement according to claim 4, characterized in that, The noise type classification loss is calculated using the cross-entropy loss function, and the noise intensity regression loss and the speech signal gain loss are calculated using the mean square error loss function.

6. The intelligent hearing aid method based on spectrum enhancement according to claim 5, characterized in that, The specific steps of performing multiple iterations and hyperparameter tuning of the spectrum enhancement model until model convergence and then evaluating the model are as follows: According to the defined model training method, the spectrum enhancement model is trained iteratively multiple times on the training set; Set a loss threshold and a maximum number of iterations. After each training cycle, calculate the loss on the validation set. If the validation set loss gradually decreases and the change is less than the preset loss threshold in several consecutive training iterations, the model is considered to have converged. If the validation set loss gradually increases or remains unchanged and the change is greater than the preset loss threshold in several consecutive training iterations, the model is considered to have not converged, and the model and hyperparameters are adjusted. If the model has not converged after reaching the maximum number of iterations, the model and hyperparameters are adjusted. The trained model is loaded onto the test set, and the model performance is judged by calculating the objective speech quality (PESQ) score of the speech signal. When the PESQ score is greater than or equal to 3.0 and less than 3.5, the model performance is considered satisfactory; when the PESQ score is greater than or equal to 3.5 and less than 4, the model performance is considered good; when the PESQ score is greater than or equal to 4, the model performance is considered excellent; when the PESQ score is less than 3.5, the model performance is considered unsatisfactory, the model and hyperparameters are adjusted, and the iteration is restarted.

7. The intelligent hearing aid method based on spectrum enhancement according to claim 6, characterized in that, The hyperparameter tuning employs a Bayesian optimization algorithm.

8. The intelligent hearing aid method based on spectrum enhancement according to claim 7, characterized in that, The specific steps for performing spectrum enhancement processing and outputting are as follows: Convert real-time audio signals into a spectrogram; Input the spectrogram into the spectrum enhancement model, calculate the speech signal gain based on the noise type and noise intensity, and output the enhanced spectrogram. Perform an inverse short-time Fourier transform on the gained spectrum to output the audio signal.

9. A smart hearing aid device based on spectrum enhancement, characterized in that, The method for performing the intelligent hearing aid method based on spectrum enhancement as described in any one of claims 1-8 includes: The signal input module is used to acquire audio signals and perform preprocessing operations on the audio signals; The spectrum conversion and data annotation module is used to convert the input audio signal into a spectrum and perform data annotation. The model training module is used to build and iteratively fine-tune the spectrum enhancement model; The spectrum gain calculation module is used to calculate the gain of the speech signal based on the noise type and noise intensity, and output the spectrum after gain. The signal output module is used to convert the gained spectrogram into an audio signal for output.

Images