Intelligent hearing aid method and device based on spectrum enhancement

Abstract

The invention provides an intelligent hearing aid method and device based on spectrum enhancement, and relates to the technical field of voice signal processing, and the method comprises the following steps: collecting an audio signal, and carrying out the preprocessing of the audio signal; converting the audio signal into a spectrogram, performing data annotation, and dividing the spectrogram into a training set, a verification set and a test set; creating a spectrum enhancement model and inputting a spectrogram of the training set for model training; performing multiple iterations on the spectrum enhancement model, performing hyper-parameter tuning until the model converges, and evaluating the model; and inputting a real-time audio signal, performing spectrum enhancement processing, and outputting. According to the invention, the frequency spectrum enhancement model is constructed and trained to perform frequency spectrum enhancement processing on the input audio signal, the frequency spectrum gain of the voice signal can be adaptively adjusted according to different noise types and noise intensities on different frequency bands, the quality of the voice signal is effectively improved, and the hearing aid effect is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of speech signal processing, and in particular to an intelligent hearing aid method and device based on spectrum enhancement. Background Art

[0002] In recent years, with increasing public awareness of hearing impairments, intelligent hearing aid technology has rapidly developed and become a key tool for improving the quality of life for people with hearing impairments. Traditional hearing aids amplify sound signals to improve the audibility of speech signals. However, due to interference from ambient noise, the effectiveness of traditional hearing aids often fails to meet user needs in complex noisy environments. To this end, modern intelligent hearing aid technology has gradually introduced methods based on signal processing and deep learning. These algorithms optimize audio signals, improving the clarity and intelligibility of speech signals, and thus improving the auditory experience for people with hearing impairments.

[0003] Spectral enhancement methods are becoming increasingly important for improving hearing aid technology because they can more accurately process speech signals in the frequency domain. Through spectral enhancement, the system can suppress background noise, enhance the audibility of speech signals in noisy environments, and improve speech clarity, significantly enhancing the user's listening experience.

[0004] While existing deep learning-based intelligent hearing aid methods have improved hearing aid effectiveness to a certain extent, they still have some limitations. Existing methods have poor adaptability to different types of noise, especially in environments with large variations in noise type and intensity, resulting in less than ideal spectral enhancement.

[0005] How to solve the above technical problems is the subject faced by the present invention. Summary of the Invention

[0006] In order to address the deficiencies of the prior art, the present invention provides an intelligent hearing aid method and device that adaptively adjusts spectral gain according to different noise types and noise intensities, thereby effectively improving the hearing aid effect.

[0007] The technical solution adopted by the present invention to solve the technical problem is: the present invention provides an intelligent hearing aid method based on spectrum enhancement, comprising the following steps: Collect audio signals and pre-process the audio signals; Convert audio signals into spectrograms, perform data annotation, and divide them into training sets, validation sets, and test sets; The training set, validation set, and test set comprise 70%, 20%, and 10% of the collected audio signal, respectively. The training set is the dataset used to train the model. The model adjusts its parameters by learning the relationships between sample features and labels in the training set, enabling it to fit patterns in the training data and make predictions on new data. The validation set is used to evaluate model performance during training and adjust model hyperparameters (such as the learning rate, number of network layers, and number of neurons). The test set is used to ultimately evaluate the model's generalization ability, that is, its ability to adapt to new data that differs from the training data.

[0008] Create a spectrum enhancement model and input the spectrum graph of the training set for model training; Perform multiple iterations of the spectrum enhancement model, tune hyperparameters until the model converges, and evaluate the model; Input real-time audio signal, perform spectrum enhancement processing and output.

[0009] Furthermore, the audio signal is a speech signal containing noises of different types and intensities.

[0010] Noise types include mechanical noise, electrical noise, environmental background noise and echo noise; Noise intensity includes weak noise, with a signal-to-noise ratio higher than +5dB; medium noise, with a signal-to-noise ratio between -10dB and +5dB; and strong noise, with a signal-to-noise ratio lower than -10dB.

[0011] Furthermore, the preprocessing includes silent cropping, normalization, framing and dynamic windowing.

[0012] By dividing the signal into frames, calculating the energy of each frame, and setting an energy threshold, when the energy is less than the preset threshold, this frame is considered to be a silent segment. The silent segments in the training audio signal are cropped to reduce the amount of model calculation and improve the model convergence speed.

[0013] Normalization eliminates amplitude inconsistencies caused by differences in audio source intensity, thereby avoiding training bias caused by volume differences.

[0014] Based on the sampling rate, predetermined frame length, and frame shift, the start and end positions of each frame are calculated. The corresponding signal segments are then extracted to achieve signal framing. Each frame typically contains tens of milliseconds of audio signal. Audio signals are typically non-stationary. Framing allows analysis within a shorter time window, enabling the model to capture more detailed time-frequency features and effectively improve analysis accuracy.

[0015] By multiplying each frame signal by a dynamic window function, a windowed frame signal is obtained to reduce discontinuities at the frame boundaries, reduce spectrum leakage during signal processing, and improve the spectrum representation quality of the signal, thereby making the spectrum graph smoother and more accurate, which is beneficial to subsequent spectrum enhancement and feature extraction.

[0016] Furthermore, converting the audio signal into a spectrogram includes performing short-time Fourier transform on the preprocessed audio signal, arranging the spectrum data into a spectrogram, and performing logarithmic transformation and scaling processing on the spectrogram.

[0017] The audio signal is converted from a time domain signal to a frequency domain signal through STFT, and the spectrum data of each frame (usually the amplitude spectrum) is stacked in time order to obtain a matrix. The rows of the matrix represent the frequency , columns represent time , the value of each cell represents the amplitude information of the speech signal and noise signal at the corresponding time and frequency, and finally forms a two-dimensional spectrum diagram.

[0018] The logarithmic transformation calculation formula is as follows: , Where, is the amplitude in the original spectrum, The logarithmic transformation is the amplitude of the spectrum after the logarithmic transformation. Since the amplitude of the spectrum may contain zero values ​​(for example, in silence), directly performing a logarithmic transformation on zero values ​​can lead to computational instability. Therefore, a small constant of 1 is added to avoid zero values ​​in the logarithm. The logarithmic transformation compresses the dynamic range of the spectrum, creating a more balanced contrast between high-energy and low-energy components. This helps highlight the characteristics of the low-energy components, improving noise suppression and speech recognition.

[0019] The scale of the spectrum is adjusted to a uniform range through scaling processing, making it more suitable for subsequent processing and avoiding poor subsequent processing effects due to excessive amplitude differences.

[0020] Furthermore, the spectrum enhancement model adopts a CNN convolutional neural network model.

[0021] CNNs have advantages in processing two-dimensional data (such as images and spectrograms) and are widely used in speech enhancement and audio signal processing tasks. The input signal of the present invention is a spectrogram, rather than a raw continuously sampled signal. Converting the audio signal from the time domain to a spectrogram in the frequency domain via the Short-Time Fourier Transform (STFT) better preserves the frequency characteristics of the speech signal. The spectrogram is a two-dimensional structured data, and CNNs can extract local correlation features in time and frequency through convolution operations.

[0022] The creation of the spectrum enhancement model and inputting the spectrum graph of the training set for model training is specifically as follows: After the training audio signal is input into the spectrum enhancement model, forward propagation is performed to extract the type and intensity distribution of noise in the training audio signal in the spectrum graph, calculate the corresponding speech signal gain, and output the gained spectrum graph.

[0023] It should be noted that during forward propagation, the model extracts the type and intensity distribution of noise from the training audio signal within the spectrogram. For example, the convolutional layer slides the convolution kernel across the spectrogram, extracting features at different locations and frequency ranges. These features can reflect both local and global information about the noise. The pooling layer reduces the dimensionality of these features, extracting key feature information while reducing computational effort and the risk of model overfitting.

[0024] The formula for calculating the speech signal gain is as follows: ; Where, is the speech signal gain, For time, is the frequency, for Time, frequency The speech signal amplitude at for Time, frequency The noise signal amplitude at is the signal-to-noise ratio, Adjust the correlation constant for the signal-to-noise ratio, for Time, frequency The dynamic adjustment factor at For custom functions, is the adjustment function based on the signal-to-noise ratio.

[0025] According to different frequencies The speech signal amplitude at , noise signal amplitude and signal-to-noise ratio , combined with the dynamic adjustment factor Calculate the appropriate gain to enhance the speech signal spectrum, improve the speech signal quality and auditory experience.

[0026] ; Where, is the noise intensity weight, is the noise type weight.

[0027] Combined with custom functions , so that the dynamic adjustment factor not only considers the noise intensity and type, but also the relative relationship between the voice signal and the noise signal amplitude, which more comprehensively reflects the gain adjustment requirements in different situations. When it is larger, Closer to 1, making The larger the noise type, the stronger the gain adjustment will be. and noise intensity weight Differentiated gain adjustments can be made according to different types of noise (such as mechanical noise, electrical noise, etc.) and different noise intensities.

[0028] It should be noted that The gain can be obtained based on statistical experimental data, typically determined by the actual gain requirements across different signal-to-noise ratio (SNR) ranges. For example, a lower weight is assigned at higher SNRs, while a higher weight is assigned at lower SNRs. γ represents the importance of different noise types to signal processing. For example, mechanical noise, electrical noise, and background noise have different impacts on speech quality. This value can be assigned through data analysis or expert experience. For example, using a manually annotated dataset, speech intelligibility under different noise types can be scored and then normalized to calculate the weight.

[0029] In the iteration of the model, and The gradient descent algorithm is used to find the optimal value for the corresponding noise type and noise intensity. Since the noise environment and signal conditions in actual applications are diverse and constantly changing, the dynamic adjustment factor enables the spectrum enhancement model to adapt to these changes. The dynamic adjustment factor combines the information of noise intensity and noise type, so that the gain adjustment does not rely on a single factor, but comprehensively considers the various characteristics of the noise and its relationship with the speech signal. In this way, while suppressing the noise, the useful components of the speech signal can be retained and enhanced as much as possible, achieving a better speech enhancement effect and listening experience. For example, for a certain type of noise (caused by If the noise level is not particularly high (but the noise level is not particularly high), the impact on the speech signal is significant. The dynamic adjustment factor will also adjust the gain accordingly to better cope with the impact of this noise on speech and achieve a balance between noise suppression and speech enhancement.

[0030] ; The size of the function value depends on the relative relationship between the amplitudes of the speech signal and the noise signal. By introducing nonlinear adjustment through the sine function, when the speech signal is relatively strong, the function value will increase, thereby enhancing the gain adjustment of the speech signal; when the noise signal is strong, the function value will decrease, reducing the gain adjustment of the speech signal.

[0031] ; According to the signal-to-noise ratio The gain adjustment amplitude is adjusted based on the size of the signal-to-noise ratio (SNR). Its contribution to overall gain varies with different SNRs. It is designed to increase gain adjustment when noise is strong (low SNR) and decrease it when voice signals are strong (high SNR). Working in conjunction with other components, it achieves the effect of dynamically adjusting gain based on the SNR.

[0032] The spectrum after gain is expressed as follows: ; Where, is the spectrum after gain, is the exponential adjustment constant, is the initial spectrum.

[0033] According to the signal-to-noise ratio , through the exponential function The results are further adjusted to prevent the exponential function from changing too quickly, which may cause the spectrum to be over-adjusted in the case of strong noise, or produce undesirable nonlinear behavior when the noise and signal strengths are comparable.

[0034] when When it is large (the speech signal is relatively strong), The value of is relatively small. is also relatively small, the exponential function The value of will be closer to 1, making The value of is relatively small, which The adjustment of is relatively small, which avoids over-enhancing the speech signal when the speech signal itself is already strong.

[0035] when When is small (noise is relatively strong), The value of is relatively large. will also increase accordingly, the exponential function The value of will be closer to 0, then The value is closer to 1, The adjustment is larger, thereby enhancing the voice signal in the frequency band with stronger noise, suppressing noise and highlighting the voice signal.

[0036] Define the loss function of the spectrum enhancement model and calculate the loss.

[0037] The loss function is defined as: ; Where, is the total loss, 、 、 is the weight coefficient, which is used to balance the contribution of different losses to model optimization and can be adjusted according to experiments; is the noise type classification loss, is the noise intensity regression loss, is the speech signal gain loss; The gradient of the loss function is calculated by the back-propagation algorithm, and the parameters of the spectrum enhancement model are updated using the gradient descent algorithm.

[0038] It should be noted that the backpropagation algorithm begins at the output layer and, using the chain rule, gradually calculates the gradient of the loss function for each layer's parameters. For example, for the weights and biases of the convolutional layer, the gradient values ​​are calculated by calculating their partial derivatives with respect to the loss function. The parameters of the spectral enhancement model are then updated using the gradient descent algorithm. The gradient descent algorithm updates the model parameters based on the calculated gradients at a specific learning rate.

[0039] Furthermore, the noise type classification loss is calculated by a cross entropy loss function, and the noise intensity regression loss and the speech signal gain loss are calculated by a mean square error loss function.

[0040] Noise type classification loss is used to distinguish noise types (such as mechanical noise, electrical noise, etc.). The cross-entropy loss function is usually used: ; in, is the true noise type label, is the predicted probability of the noise type output by the model, is the total number of samples.

[0041] Noise intensity regression loss is used to predict noise intensity, and the mean squared error (MSE) is usually used as the loss function: ; in, is the true noise intensity, is the noise intensity predicted by the model, is the total number of samples.

[0042] The speech signal enhancement loss is used to evaluate the difference between the enhanced speech signal and the original signal. It is calculated using the mean square error of the spectral amplitude values ​​before and after enhancement: ; in, is the amplitude value before enhancement, is the enhanced amplitude value, is the number of frequency components.

[0043] Furthermore, the spectrum enhancement model is iterated multiple times, hyperparameters are tuned until the model converges, and the model is evaluated according to the PESQ score. Specifically, According to the defined model training method, the spectrum enhancement model is trained iteratively multiple times on the training set.

[0044] By continuously repeating the process of forward propagation, loss calculation, backpropagation and parameter update, the model parameters are gradually optimized, so that the performance of the model on the training set is continuously improved, and ultimately it can better perform spectral enhancement processing on new audio signals.

[0045] Set the loss threshold and the maximum number of iterations. After each training cycle, calculate the loss on the validation set. If the validation set loss gradually decreases and the change is less than the preset loss threshold in several consecutive iterative trainings, the model is judged to have converged. If the validation set loss gradually increases or remains unchanged and the change is greater than the preset loss threshold in several consecutive iterative trainings, the model is judged to have not converged, and the model and hyperparameters are adjusted. If the model has not converged after the maximum number of iterations, the model and hyperparameters are adjusted.

[0046] By setting a maximum number of iterations, you can limit training time and prevent the training process from continuing indefinitely. This helps control the consumption of computing resources (such as CPU and GPU usage and memory usage). It also prevents overfitting caused by excessive training. Furthermore, if there are problems with the algorithm itself or the data is not suitable for the model, the algorithm may not converge to a stable solution. For example, when using an inappropriate learning rate for gradient descent, the algorithm may oscillate back and forth in the parameter space and fail to converge. In this case, the maximum number of iterations can limit this ineffective iterative process.

[0047] The trained model is loaded onto the test set, and the performance of the model is evaluated by calculating the objective speech quality (PESQ) score of the speech signal. To perform this calculation, the original audio signal and the gain-processed audio signal are first aligned at their starting and ending points. The original and gain-processed signals are then divided into several time frames (typically 20-30 milliseconds in length, with 50%-75% overlap). Then, based on the standard PESQ algorithm developed by the International Telecommunication Union (ITU), the auditory perception model, which is based on the human ear conduction model, is loaded. A filter bank is used to map the signal's spectral changes to outputs similar to human ear perception. The auditory perception model extracts features from each frame, including spectral distortion and temporal changes. Based on this frame analysis, a distortion metric similar to human ear perception is calculated by comparing the spectral and temporal characteristics of the original and gain-processed audio signals. Finally, the PESQ algorithm integrates the perceived quality of the entire audio signal to generate an overall PESQ score.

[0048] When the PESQ score is greater than or equal to 3.0 and less than 3.5, the model performance is considered to be up to standard; when the PESQ score is greater than or equal to 3.5 and less than 4, the model performance is considered to be good; when the PESQ score is greater than or equal to 4, the model performance is considered to be excellent; when the PESQ score is less than 3.5, the model performance is considered to be substandard, and the model and hyperparameters should be adjusted and iterated again.

[0049] The PESQ score typically ranges from 4-5: Excellent, near-perfect sound quality. 3-4: Good, clear sound quality with only minor artifacts. 2-3: Fair, acceptable sound quality with noticeable artifacts. 1-2: Poor, poor sound quality with limited clarity. 0-1: Very poor, unintelligible and barely acceptable. A PESQ score of 3.0 or higher indicates high-quality speech signals, meeting most speech clarity requirements.

[0050] Furthermore, the hyperparameter tuning adopts a Bayesian optimization algorithm.

[0051] It should be noted that in machine learning models, hyperparameters are usually pre-set parameters during the training process, such as learning rate and batch size. These parameters themselves are not directly reflected in the data representation, but indirectly determine the performance of the model by affecting the model training process and convergence behavior.

[0052] The significance of hyperparameter adjustment: The "hyperparameter adjustment" mentioned in this article is intended to optimize the model training process and performance, rather than adjusting specific input or output data. Therefore, although hyperparameters don't have a direct data representation, adjusting their value range can improve model performance and convergence speed, a common method for machine learning model optimization.

[0053] Furthermore, the input real-time audio signal is subjected to spectrum enhancement processing and the output is specifically: Convert real-time audio signals into spectrograms.

[0054] To achieve the conversion from time domain to frequency domain, short-time Fourier transform (STFT) is used, which divides the audio signal into multiple time segments, and performs Fourier transform on each segment and merges them to obtain a two-dimensional spectrogram, with the horizontal axis being time, the vertical axis being frequency, and the color representing the amplitude.

[0055] The spectrogram is input into the spectrum enhancement model, the speech signal gain is calculated according to the noise type and noise intensity, and the gained spectrogram is output.

[0056] After the spectrogram is input into the spectrum enhancement model, the convolutional neural network analyzes the input spectrogram, identifies the noise type and noise intensity in different frequency bands, and calculates the gain of the speech signal in different frequency bands through the defined gain function based on the noise type and noise intensity. The calculated gain is multiplied by the spectrogram, the spectrogram is gain-processed, and finally the gained spectrogram is output.

[0057] Perform inverse short-time Fourier transform on the amplified spectrum to output the audio signal.

[0058] The spectrogram after gain processing is converted back to the time domain through an inverse short-time Fourier transform (ISTFT). The purpose of this process is to restore the audio signal to make it suitable for playback through hearing aids.

[0059] The present invention also provides an intelligent hearing aid device based on spectrum enhancement, comprising: Signal input module: collects audio signals and performs pre-processing operations on the audio signals; Spectrum conversion and data annotation module: converts the input audio signal into a spectrum graph and performs data annotation; Model training module: defines model training methods, model parameters, and hyperparameters; Spectral gain calculation module: calculates the speech signal gain according to the noise type and noise intensity, and outputs the gained spectrum; Signal output module: converts the amplified spectrum into audio signal output.

[0060] The beneficial effects of the present invention are: adaptively adjusting the spectral gain according to different noise types and noise intensities, effectively improving the hearing aid effect. The silent segments in the training audio signal are cropped to reduce the model calculation amount and improve the model convergence speed. The amplitude inconsistency problem caused by differences in audio source intensity is eliminated through normalization, and training deviation caused by volume differences is avoided. Through framing and dynamic windowing, the model can capture more detailed time-frequency features, reduce discontinuities at frame boundaries, and reduce spectral leakage during signal processing, thereby improving the spectral representation quality of the signal. The dynamic range of the spectral amplitude is reduced by logarithmic transformation, so that the difference between high-intensity signals and low-intensity signals is not too large, avoiding excessively strong signals from drowning out weaker signals, thereby improving the robustness and accuracy of the speech enhancement algorithm. The spectrogram is scaled to a uniform scale to avoid learning deviations caused by different data magnitudes. The distribution of noise type and noise intensity on the spectrogram is extracted through forward propagation of the convolutional neural network, and the gain output is calculated accordingly, achieving adaptive adjustment for different noise types and noise intensities. The gradient descent algorithm is used to update model parameters and the Bayesian optimization algorithm is used to tune hyperparameters to further enhance the convergence speed and accuracy of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 A diagram showing the steps of the method of the present invention.

[0062] Figure 2 It is a device module diagram of the present invention. DETAILED DESCRIPTION

[0063] In order to clearly illustrate the technical features of this solution, the solution is described below through specific implementation methods.

[0064] Example 1: See also Figure 1 As shown, this embodiment provides an intelligent hearing aid method based on spectrum enhancement, including the following steps: S1. Collect audio signals and pre-process the audio signals.

[0065] The audio signal is a speech signal containing noise of different types and intensities.

[0066] It should be noted that noise types include mechanical noise, electrical noise, environmental background noise and echo noise; noise intensity includes weak noise, with a signal-to-noise ratio higher than +5dB; medium noise, with a signal-to-noise ratio between -10dB and +5dB; and strong noise, with a signal-to-noise ratio lower than -10dB.

[0067] Preprocessing includes silent cropping, normalization, framing and dynamic windowing.

[0068] By dividing the signal into frames, calculating the energy of each frame, and setting an energy threshold, when the energy is less than the preset threshold, this frame is considered to be a silent segment. The silent segments in the training audio signal are cropped to reduce the amount of model calculation and improve the model convergence speed.

[0069] Normalization eliminates amplitude inconsistencies caused by differences in audio source intensity, thereby avoiding training bias caused by volume differences.

[0070] Based on the sampling rate, predetermined frame length, and frame shift, the start and end positions of each frame are calculated. The corresponding signal segments are then extracted to achieve signal framing. Each frame typically contains tens of milliseconds of audio signal. Audio signals are typically non-stationary. Framing allows analysis within a shorter time window, enabling the model to capture more detailed time-frequency features and effectively improve analysis accuracy.

[0071] By multiplying each frame signal by a dynamic window function, a windowed frame signal is obtained to reduce discontinuities at the frame boundaries, reduce spectrum leakage during signal processing, and improve the spectrum representation quality of the signal, thereby making the spectrum graph smoother and more accurate, which is beneficial to subsequent spectrum enhancement and feature extraction.

[0072] S2. Convert the audio signal into a spectrogram, perform data annotation, and divide it into training set, validation set, and test set.

[0073] Converting the audio signal into a spectrogram includes performing short-time Fourier transform on the preprocessed audio signal, arranging the spectrum data into a spectrogram, and performing logarithmic transformation and scaling on the spectrogram.

[0074] The audio signal is converted from a time domain signal to a frequency domain signal through STFT, and the spectrum data of each frame (usually the amplitude spectrum) is stacked in time order to obtain a matrix. The rows of the matrix represent the frequency , columns represent time , the value of each cell represents the amplitude information of the speech signal and noise signal at the corresponding time and frequency, and finally forms a two-dimensional spectrum diagram.

[0075] It should be noted that the logarithmic transformation calculation formula is as follows: ; Where, is the amplitude in the original spectrum, The logarithmic transformation is the amplitude of the spectrum after the logarithmic transformation. Since the amplitude of the spectrum may contain zero values ​​(for example, in silence), directly performing a logarithmic transformation on zero values ​​can lead to computational instability. Therefore, a small constant of 1 is added to avoid zero values ​​in the logarithm. The logarithmic transformation compresses the dynamic range of the spectrum, creating a more balanced contrast between high-energy and low-energy components. This helps highlight the characteristics of the low-energy components, improving noise suppression and speech recognition.

[0076] The scale of the spectrum is adjusted to a uniform range through scaling processing, making it more suitable for subsequent processing and avoiding poor subsequent processing effects due to excessive amplitude differences.

[0077] The data annotation content includes speech signal, noise signal, noise type, and noise intensity.

[0078] Logarithmic transformation effectively reduces the dynamic range of the spectrum amplitude, so that the difference between high-intensity signals and low-intensity signals is not too large, reducing the difficulty of model training and preventing overly strong signals from drowning out weaker signals, thereby improving the robustness and accuracy of the speech enhancement algorithm.

[0079] Scale the spectrogram to a uniform scale to avoid learning bias caused by different data magnitudes.

[0080] The training set, validation set, and test set comprise 70%, 20%, and 10% of the collected audio signal, respectively. The training set is the dataset used to train the model. The model learns the relationship between sample features and labels in the training set to adjust its parameters, enabling it to fit patterns in the training data and make predictions on new data. The validation set is used to evaluate model performance during training and adjust model hyperparameters (such as the learning rate, number of network layers, and number of neurons). The test set is used to ultimately evaluate the model's generalization ability, that is, its ability to adapt to new data that differs from the training data.

[0081] S3. Create a spectrum enhancement model and input the spectrum graph of the training set for model training.

[0082] The spectrum enhancement model uses a convolutional neural network (CNN) model. CNNs are advantageous for processing two-dimensional data (such as images and spectrograms) and are widely used in speech enhancement and audio signal processing tasks. The input signal of this method is a spectrogram, rather than a raw continuously sampled signal. The short-time Fourier transform (STFT) converts the audio signal from the time domain to a spectrogram in the frequency domain, better preserving the frequency characteristics of the speech signal. A spectrogram is a two-dimensional structured data type, and CNNs can extract local correlation features in time and frequency through convolution operations.

[0083] Create a spectrum enhancement model and input the spectrogram of the training audio signal for model training as follows: S30: After the training audio signal is input into the spectrum enhancement model, forward propagation is performed to extract the type and intensity distribution of noise in the training audio signal in the spectrum graph, calculate the corresponding speech signal gain, and output the gained spectrum graph.

[0084] It should be noted that during forward propagation, the model extracts the type and intensity distribution of noise from the training audio signal within the spectrogram. For example, the convolutional layer slides the convolution kernel across the spectrogram, extracting features at different locations and frequency ranges. These features can reflect both local and global information about the noise. The pooling layer reduces the dimensionality of these features, extracting key feature information while reducing computational effort and the risk of model overfitting.

[0085] The formula for calculating the speech signal gain is as follows: ; Where, is the speech signal gain, For time, is the frequency, for Time, frequency The speech signal amplitude at for Time, frequency The noise signal amplitude at is the signal-to-noise ratio, Adjust the correlation constant for the signal-to-noise ratio, for Time, frequency The dynamic adjustment factor at For custom functions, is the adjustment function based on the signal-to-noise ratio.

[0086] According to different frequencies The speech signal amplitude at , noise signal amplitude and signal-to-noise ratio , combined with the dynamic adjustment factor Calculate the appropriate gain to enhance the speech signal spectrum, improve the speech signal quality and auditory experience.

[0087] ; Where, is the noise intensity weight, is the noise type weight.

[0088] Combined with custom functions , so that the dynamic adjustment factor not only considers the noise intensity and type, but also the relative relationship between the voice signal and the noise signal amplitude, which more comprehensively reflects the gain adjustment requirements in different situations. When it is larger, Closer to 1, making The larger the noise type, the stronger the gain adjustment will be. and noise intensity weight Differentiated gain adjustments can be made according to different types of noise (such as mechanical noise, electrical noise, etc.) and different noise intensities.

[0089] It should be noted that The gain can be obtained based on statistical experimental data, typically determined by the actual gain requirements across different signal-to-noise ratio (SNR) ranges. For example, a lower weight is assigned at higher SNRs, while a higher weight is assigned at lower SNRs. γ represents the importance of different noise types to signal processing. For example, mechanical noise, electrical noise, and background noise have different impacts on speech quality. This value can be assigned through data analysis or expert experience. For example, using a manually annotated dataset, speech intelligibility under different noise types can be scored and then normalized to calculate the weight.

[0090] Furthermore, in the iteration of the model, and The gradient descent algorithm is used to find the optimal value for the corresponding noise type and noise intensity. Since the noise environment and signal conditions in actual applications are diverse and constantly changing, the dynamic adjustment factor enables the spectrum enhancement model to adapt to these changes. The dynamic adjustment factor combines the information of noise intensity and noise type, so that the gain adjustment does not rely on a single factor, but comprehensively considers the various characteristics of the noise and its relationship with the speech signal. In this way, while suppressing the noise, the useful components of the speech signal can be retained and enhanced as much as possible, achieving a better speech enhancement effect and listening experience. For example, for a certain type of noise (caused by If the noise level is not particularly high (but the noise level is not particularly high), the impact on the speech signal is significant. The dynamic adjustment factor will also adjust the gain accordingly to better cope with the impact of this noise on speech and achieve a balance between noise suppression and speech enhancement.

[0091] ; The size of the function value depends on the relative relationship between the amplitudes of the speech signal and the noise signal. By introducing nonlinear adjustment through the sine function, when the speech signal is relatively strong, the function value will increase, thereby enhancing the gain adjustment of the speech signal; when the noise signal is strong, the function value will decrease, reducing the gain adjustment of the speech signal.

[0092] ; According to the signal-to-noise ratio The gain adjustment amplitude is adjusted based on the size of the signal-to-noise ratio (SNR). Its contribution to overall gain varies with different SNRs. It is designed to increase gain adjustment when noise is strong (low SNR) and decrease it when voice signals are strong (high SNR). Working in conjunction with other components, it achieves the effect of dynamically adjusting gain based on the SNR.

[0093] The spectrum after gain is expressed as follows: ; Where, is the spectrum after gain, is the exponential adjustment constant, is the initial spectrum.

[0094] According to the signal-to-noise ratio , through the exponential function The results are further adjusted to prevent the exponential function from changing too quickly, which may cause the spectrum to be over-adjusted in the case of strong noise, or produce undesirable nonlinear behavior when the noise and signal strengths are comparable.

[0095] when When it is large (the speech signal is relatively strong), The value of is relatively small. is also relatively small, the exponential function The value of will be closer to 1, making The value of is relatively small, which The adjustment of is relatively small, which avoids over-enhancing the speech signal when the speech signal itself is already strong.

[0096] when When is small (noise is relatively strong), The value of is relatively large. will also increase accordingly, the exponential function The value of will be closer to 0, then The value is closer to 1, The adjustment is larger, thereby enhancing the voice signal in the frequency band with stronger noise, suppressing noise and highlighting the voice signal.

[0097] S31. Define the loss function of the spectrum enhancement model and calculate the loss.

[0098] The loss function is defined as: ; Where, is the total loss, 、 、 is the weight coefficient, which is used to balance the contribution of different losses to model optimization and can be adjusted according to experiments; It is the noise type classification loss, which is used to distinguish the noise types (such as mechanical noise, electrical noise, etc.). The cross-entropy loss function is usually used: ; in, is the true noise type label, is the predicted probability of the noise type output by the model, is the total number of samples.

[0099] is the noise intensity regression loss, which is used to predict the noise intensity. The mean squared error (MSE) is usually used as the loss function: ; in, is the true noise intensity, is the noise intensity predicted by the model, is the total number of samples.

[0100] is the speech signal enhancement loss, which is used to evaluate the difference between the enhanced speech signal and the original signal. It is calculated using the mean square error of the spectrum amplitude values ​​before and after enhancement: ; in, is the amplitude value before enhancement, is the enhanced amplitude value, is the number of frequency components.

[0101] S32. Calculate the gradient of the loss function through the back-propagation algorithm, and use the gradient descent algorithm to update the parameters of the spectrum enhancement model.

[0102] It should be noted that the backpropagation algorithm begins at the output layer and, using the chain rule, gradually calculates the gradient of the loss function for each layer's parameters. For example, for the weights and biases of the convolutional layer, the gradient values ​​are calculated by calculating their partial derivatives with respect to the loss function. The parameters of the spectral enhancement model are then updated using the gradient descent algorithm. The gradient descent algorithm updates the model parameters based on the calculated gradients at a specific learning rate.

[0103] S4. Perform multiple iterations of the spectrum enhancement model, tune the hyperparameters until the model converges, and evaluate the model based on the PESQ score.

[0104] According to the defined model training method, the spectrum enhancement model is trained iteratively multiple times on the training set.

[0105] By continuously repeating the process of forward propagation, loss calculation, backpropagation and parameter update, the model parameters are gradually optimized, so that the performance of the model on the training set is continuously improved, and ultimately it can better perform spectral enhancement processing on new audio signals.

[0106] Set the loss threshold and the maximum number of iterations. After each training cycle, calculate the loss on the validation set. If the validation set loss gradually decreases and the change is less than the preset loss threshold in several consecutive iterative trainings, the model is judged to have converged. If the validation set loss gradually increases or remains unchanged and the change is greater than the preset loss threshold in several consecutive iterative trainings, the model is judged to have not converged, and the model and hyperparameters are adjusted. If the model has not converged after the maximum number of iterations, the model and hyperparameters are adjusted.

[0107] By setting a maximum number of iterations, you can limit training time and prevent the training process from continuing indefinitely. This helps control the consumption of computing resources (such as CPU and GPU usage and memory usage). It also prevents overfitting caused by excessive training. Furthermore, if there are problems with the algorithm itself or the data is not suitable for the model, the algorithm may not converge to a stable solution. For example, when using an inappropriate learning rate for gradient descent, the algorithm may oscillate back and forth in the parameter space and fail to converge. In this case, the maximum number of iterations can limit this ineffective iterative process.

[0108] The trained model is loaded onto the test set, and the model performance is judged by calculating the objective speech quality (PESQ) score of the speech signal. During the calculation, the starting and ending points of the original audio signal and the gain-processed audio signal are first synchronized, and the original audio signal and the gain-processed audio signal are respectively divided into several time frames (usually each frame is 20-30 milliseconds long, with an overlap of 50%-75%). Then, the PESQ algorithm loads the auditory perception model based on the human ear conduction model, and uses a filter bank to map the signal's spectral changes to an output similar to human ear perception. The auditory perception model extracts the features of each frame, including spectral distortion and time domain changes. Based on the analysis of each frame, a distortion metric similar to human ear perception is calculated by comparing the spectral and time domain features of the original audio signal and the gain-processed audio signal. Ultimately, the PESQ algorithm integrates the perceptual quality of the entire audio signal to generate an overall PESQ score.

[0109] When the PESQ score is greater than or equal to 3.0 and less than 3.5, the model performance is considered to be up to standard; when the PESQ score is greater than or equal to 3.5 and less than 4, the model performance is considered to be good; when the PESQ score is greater than or equal to 4, the model performance is considered to be excellent; when the PESQ score is less than 3.5, the model performance is considered to be substandard, and the model and hyperparameters should be adjusted and iterated again.

[0110] The PESQ score typically ranges from 4-5: Excellent, near-perfect sound quality. 3-4: Good, clear sound quality with only minor artifacts. 2-3: Fair, acceptable sound quality with noticeable artifacts. 1-2: Poor, poor sound quality with limited clarity. 0-1: Very poor, unintelligible and barely acceptable. A PESQ score of 3.0 or higher indicates high-quality speech signals, meeting most speech clarity requirements.

[0111] Hyperparameter tuning uses the Bayesian optimization algorithm.

[0112] It should be noted that in machine learning models, hyperparameters are usually pre-set parameters during the training process, such as learning rate and batch size. These parameters themselves are not directly reflected in the data representation, but indirectly determine the performance of the model by affecting the model training process and convergence behavior.

[0113] The significance of hyperparameter adjustment: The "hyperparameter adjustment" mentioned in this article is intended to optimize the model training process and performance, rather than adjusting specific input or output data. Therefore, although hyperparameters don't have a direct data representation, adjusting their value range can improve model performance and convergence speed, a common method for machine learning model optimization.

[0114] S5. Input the real-time audio signal, perform spectrum enhancement processing on it, and output it.

[0115] S50: convert the real-time audio signal into a spectrum graph.

[0116] To achieve the conversion from time domain to frequency domain, short-time Fourier transform (STFT) is used, which divides the audio signal into multiple time segments, and performs Fourier transform on each segment and merges them to obtain a two-dimensional spectrogram, with the horizontal axis being time, the vertical axis being frequency, and the color representing the amplitude.

[0117] S51. Input the spectrogram into a spectrum enhancement model, calculate the speech signal gain according to the noise type and noise intensity, and output the gained spectrogram.

[0118] After the spectrogram is input into the spectrum enhancement model, the convolutional neural network analyzes the input spectrogram, identifies the noise type and noise intensity in different frequency bands, and calculates the gain of the speech signal in different frequency bands through the defined gain function based on the noise type and noise intensity. The calculated gain is multiplied by the spectrogram, the spectrogram is gain-processed, and finally the gained spectrogram is output.

[0119] S52: Perform inverse short-time Fourier transform on the amplified spectrum graph to output an audio signal.

[0120] The spectrogram after gain processing is converted back to the time domain through an inverse short-time Fourier transform (ISTFT). The purpose of this process is to restore the audio signal to make it suitable for playback through hearing aids.

[0121] Example 2: See also Figure 2 As shown, this embodiment provides an intelligent hearing aid device based on spectrum enhancement, including a signal input module, a spectrum conversion and data annotation module, a model training module, a spectrum gain calculation module and a signal output module; Signal input module: collects audio signals and performs pre-processing operations on the audio signals; Spectrum conversion and data annotation module: converts the input audio signal into a spectrum graph and performs data annotation; Model training module: defines model training methods, model parameters, and hyperparameters; Spectral gain calculation module: calculates the speech signal gain according to the noise type and noise intensity, and outputs the gained spectrum; Signal output module: converts the amplified spectrum into audio signal output.

[0122] Example 3: This embodiment provides an intelligent hearing aid method based on spectrum enhancement. This method is scientifically demonstrated by calculating the PESQ scores of speech signals before and after spectrum enhancement for different noise types and noise intensities. The method aims to verify the effectiveness of improving hearing aid performance. The method includes the following steps: S1. Collect audio signals and pre-process the audio signals.

[0123] S2. Convert the audio signal into a spectrogram, perform data annotation, and divide it into training set, validation set, and test set.

[0124] S3. Create a spectrum enhancement model and input the spectrum graph of the training set for model training.

[0125] S4. Perform multiple iterations of the spectrum enhancement model, tune the hyperparameters until the model converges, and evaluate the model.

[0126] S5. Input the real-time audio signal, perform spectrum enhancement processing on it, and output it.

[0127] S50: convert the real-time audio signal into a spectrum graph.

[0128] To achieve the conversion from time domain to frequency domain, short-time Fourier transform (STFT) is used, which divides the audio signal into multiple time segments, and performs Fourier transform on each segment and merges them to obtain a two-dimensional spectrogram, with the horizontal axis being time, the vertical axis being frequency, and the color representing the amplitude.

[0129] S51. Input the spectrogram into a spectrum enhancement model, calculate the speech signal gain according to the noise type and noise intensity, and output the gained spectrogram.

[0130] After the spectrogram is input into the spectrum enhancement model, the convolutional neural network analyzes the input spectrogram, identifies the noise type and noise intensity in different frequency bands, and calculates the gain of the speech signal in different frequency bands through the defined gain function based on the noise type and noise intensity. The calculated gain is multiplied by the spectrogram, the spectrogram is gain-processed, and finally the gained spectrogram is output.

[0131] S52: Perform inverse short-time Fourier transform on the amplified spectrum graph to output an audio signal.

[0132] The spectrogram after gain processing is converted back to the time domain through an inverse short-time Fourier transform (ISTFT). The purpose of this process is to restore the audio signal to make it suitable for playback through hearing aids.

[0133] The following examples illustrate the effectiveness of this method using audio signals containing mechanical noise and environmental background noise of varying intensities. The specific data are shown in Tables 1 and 2:

[0134] Table 1 Speech signal quality after spectrum enhancement of mechanical noise of different intensities

[0135] Table 2 Speech signal quality after spectrum enhancement of background noise at different intensities From the data in Tables 1 and 2, it can be seen that after the audio signal is input and the spectrum enhancement model is trained, the PESQ score of the speech signal is significantly improved after gain processing compared with the PESQ score of the speech signal before gain processing. The PESQ scores of speech signals with different noise types and noise intensities are all increased to above 3.5, effectively improving the quality of noisy speech signals, enabling them to meet the speech signal quality requirements in most scenarios, and significantly improving the hearing aid effect in complex noisy environments.

[0136] Technical features not described in the present invention can be achieved through or by adopting existing technologies and will not be described in detail here. Of course, the above description is not a limitation of the present invention, and the present invention is not limited to the above examples. Changes, modifications, additions or substitutions made by ordinary technicians in this technical field within the essential scope of the present invention should also fall within the scope of protection of the present invention.

Claims

1. An intelligent hearing aid method based on spectrum enhancement, characterized in that: The following steps are involved: Collect audio signals and pre-process the audio signals; Convert audio signals into spectrograms, perform data annotation, and divide them into training sets, validation sets, and test sets; Create a spectrum enhancement model and input the spectrum graph of the training set for model training; Perform multiple iterations of the spectrum enhancement model, tune hyperparameters until the model converges, and evaluate the model based on the PESQ score; Input real-time audio signal, perform spectrum enhancement processing and output.

2. The intelligent hearing aid method based on spectrum enhancement according to claim 1, characterized in that: The audio signal is a speech signal containing noise of different types and intensities; Noise types include mechanical noise, electrical noise, environmental background noise and echo noise; Noise intensity includes weak noise, signal-to-noise ratio higher than +5dB; medium noise, signal-to-noise ratio within -10dB to +5dB; strong noise, signal-to-noise ratio lower than -10dB.

3. The intelligent hearing aid method based on spectrum enhancement according to claim 2, characterized in that: Preprocessing operations include silent cropping, normalization, framing, and dynamic windowing.

4. The intelligent hearing aid method based on spectrum enhancement according to claim 3, characterized in that: Converting the audio signal into a spectrogram includes performing a short-time Fourier transform on the preprocessed training audio signal, arranging the spectrum data into a spectrogram, and performing a logarithmic transformation and scaling process on the spectrogram; The data annotation includes annotation of speech signals, noise signals, noise types, and noise intensity.

5. The intelligent hearing aid method based on spectrum enhancement according to claim 4, characterized in that: The spectrum enhancement model adopts a CNN convolutional neural network model; The creation of the spectrum enhancement model and inputting the spectrum graph of the training set for model training is specifically as follows: After the training audio signal is input into the spectrum enhancement model, forward propagation is performed to extract the type and intensity distribution of noise in the training audio signal in the spectrum graph, calculate the corresponding speech signal gain, and output the gained spectrum graph; The formula for calculating the speech signal gain is as follows: ； Where, is the speech signal gain, For time, is the frequency, for Time, frequency The speech signal amplitude at for Time, frequency The noise signal amplitude at is the signal-to-noise ratio, Adjust the correlation constant for the signal-to-noise ratio, for Time, frequency The dynamic adjustment factor at For custom functions, is a regulation function based on the signal-to-noise ratio; ， Where, is the noise intensity weight, is the noise type weight; ； ； The spectrum after gain is expressed as follows: ； Where, is the spectrum after gain, is the exponential adjustment constant, is the initial spectrum diagram; Define the loss function of the spectrum enhancement model and calculate the loss; The loss function is defined as: ； Where, is the total loss, 、 、 is the weight coefficient, is the noise type classification loss, is the noise intensity regression loss, is the speech signal gain loss; The gradient of the loss function is calculated by the back-propagation algorithm, and the parameters of the spectrum enhancement model are updated using the gradient descent algorithm.

6. The intelligent hearing aid method based on spectrum enhancement according to claim 5, characterized in that: The noise type classification loss is calculated using a cross entropy loss function, and the noise intensity regression loss and the speech signal gain loss are calculated using a mean square error loss function.

7. The intelligent hearing aid method based on spectrum enhancement according to claim 6, characterized in that: The spectrum enhancement model is iterated multiple times, hyperparameters are tuned until the model converges, and the model is evaluated as follows: According to the defined model training method, the spectrum enhancement model is iteratively trained on the training set for multiple times; Set the loss threshold and the maximum number of iterations. After each training cycle, calculate the loss on the validation set. If the validation set loss gradually decreases and the change is less than the preset loss threshold in several consecutive iterative trainings, the model is considered to have converged. If the validation set loss gradually increases or remains unchanged and the change is greater than the preset loss threshold in several consecutive iterative trainings, the model is considered to have not converged, and the model and hyperparameters are adjusted. If the model has not converged after the maximum number of iterations, the model and hyperparameters are adjusted. Load the trained model on the test set and judge the model performance by calculating the objective speech quality (PESQ) score of the speech signal. When the PESQ score is greater than or equal to 3.0 and less than 3.5, the model performance is considered to be up to standard; when the PESQ score is greater than or equal to 3.5 and less than 4, the model performance is considered to be good; when the PESQ score is greater than or equal to 4, the model performance is considered to be excellent; when the PESQ score is less than 3.5, the model performance is considered to be substandard, and the model and hyperparameters should be adjusted and iterated again.

8. The intelligent hearing aid method based on spectrum enhancement according to claim 7, characterized in that: The hyperparameter tuning adopts the Bayesian optimization algorithm.

9. The intelligent hearing aid method based on spectrum enhancement according to claim 8, characterized in that: The spectrum enhancement processing and output are specifically as follows: Convert real-time audio signals into spectrograms; Input the spectrogram into the spectrum enhancement model, calculate the speech signal gain according to the noise type and noise intensity, and output the gained spectrogram; Perform inverse short-time Fourier transform on the amplified spectrum to output the audio signal.

10. An intelligent hearing aid device based on spectrum enhancement, characterized in that: include: Signal input module: collects audio signals and performs pre-processing operations on the audio signals; Spectrum conversion and data annotation module: converts the input audio signal into a spectrum graph and performs data annotation; Model training module: defines model training methods, model parameters, and hyperparameters; Spectral gain calculation module: calculates the speech signal gain according to the noise type and noise intensity, and outputs the gained spectrum; Signal output module: converts the amplified spectrum into audio signal output.

Images