Personalized voice noise reduction and enhancement method based on user specific time domain envelope reconstruction

Abstract

The invention discloses a personalized voice noise reduction and enhancement method based on user specific time domain envelope reconstruction, which relates to the field of voice signal processing, and comprises the following steps: monitoring the environmental noise level in real time when a user uses an earphone to carry out daily conversation or voice input; when the environmental noise level is monitored to be lower than a preset noise threshold value, automatically collecting the voice signal of the user at the moment as a low-noise environmental voice sample; carrying out phoneme-level segmentation on the voice sample and extracting a time domain envelope to establish a target envelope database; during real-time processing, envelope differences are compared through a dynamic time warping algorithm, local gain correction parameters are generated for dynamic gain adjustment, and enhanced real-time voice signals are obtained. According to the method, the problems of voice distortion and insufficient individuation of a traditional noise reduction method are solved.

Description

Technical Field

[0001] The present invention relates to the field of speech signal processing, and more particularly to a personalized speech noise reduction and enhancement method based on user-specific time domain envelope reconstruction. Background Art

[0002] In recent years, with the popularity of smart wearable devices, wireless headphones, and smartphones, the demand for voice communication and recording in people's daily lives has continued to increase. However, in real-world environments, background noise interference is always a significant factor affecting the quality of voice calls and voice input. Conventional voice noise reduction methods usually use a unified processing model, ignoring the differences in pronunciation and voice characteristics among individual users. Although this processing method can effectively reduce background noise, it can easily lead to voice distortion and lose the original personalized characteristics of the user's voice. Especially in personalized scenarios, the impact of voice distortion and reduced naturalness on the user experience is more obvious. Therefore, how to effectively suppress environmental noise while maintaining the naturalness and personalized characteristics of the user's voice has become a key technical problem that needs to be solved in the field of voice enhancement technology. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to provide a personalized speech noise reduction and enhancement method based on user-specific time domain envelope reconstruction to solve the problems mentioned in the background technology.

[0004] In order to achieve the above object, the present invention adopts the following technical solutions:

[0005] A personalized speech noise reduction and enhancement method based on user-specific time domain envelope reconstruction, characterized by comprising the following steps:

[0006] When the user uses the headset for daily calls or voice input, the ambient noise level is monitored in real time. When the ambient noise level is lower than the preset noise threshold, the user's voice signal at this time is automatically collected as a low-noise environment voice sample;

[0007] Performing phoneme-level segmentation on the low-noise environment speech sample, extracting the time domain envelope of each phoneme unit, and storing them as target time domain envelopes of the corresponding phonemes in a database;

[0008] When used in real time, the real-time speech signal is segmented at the phoneme level, and the real-time time domain envelope of each phoneme unit is extracted. After the real-time time domain envelope is time-aligned with the target time domain envelope of the corresponding phoneme in the database using the dynamic time warping algorithm, the difference between the real-time time domain envelope of each phoneme unit and the corresponding target time domain envelope is calculated. Based on the difference, the local gain correction parameters of each phoneme unit in the real-time speech signal are generated;

[0009] Dynamic gain adjustment is performed on each phoneme unit of the real-time speech signal according to the local gain correction parameter to obtain an enhanced real-time speech signal.

[0010] Optionally, the dynamic gain adjustment includes: first performing time domain filtering on the real-time voice signal to eliminate high-frequency peaks and low-frequency drifts in the real-time voice signal; and then using the local gain correction parameter to adjust the amplitude of the real-time voice signal after time domain filtering.

[0011] Optionally, the real-time time domain envelope is extracted using the following formula:

[0012]

[0013] Where E(n) is the short-time energy of the real-time speech at the nth moment, x(m) is the sampling value of the real-time speech signal, and L is the sliding window length.

[0014] Optionally, the difference between the real-time time envelope and the user target time envelope is calculated as follows:

[0015] D(i)=E t (i)―E r (i);

[0016] Where D(i) is the envelope difference value at the i-th moment, E t (i) is the value of the target time domain envelope at the i-th moment, E r (i) is the value of the real-time time domain envelope at the i-th moment.

[0017] Optionally, the local gain correction parameter is determined by introducing an attention mechanism, including using the attention mechanism to assign a greater weight to a time period where the real-time time domain envelope differs greatly from the target time domain envelope, so as to optimize the calculation of the local gain correction parameter.

[0018] Optionally, the method also includes a frequency domain processing channel, which uses short-time Fourier transform to perform spectral analysis on the real-time voice signal and remove environmental noise in the spectrum. After the noise reduction result of the frequency domain channel is fused with the enhancement result of the time domain channel, the final enhanced real-time voice signal is output.

[0019] Optionally, the window function of the short-time Fourier transform in the frequency domain channel is a Hamming window, the window length is set in the range of 20 milliseconds to 40 milliseconds, and the frame shift is half of the window length.

[0020] Optionally, the user target time domain envelope is normalized before storage, and the normalization is performed using the maximum value of the target time domain envelope so that the target envelope values are all between 0 and 1.

[0021] Optionally, the dynamic time warping algorithm determines the optimal alignment path between the real-time time domain envelope and the target time domain envelope based on the principle of minimum cumulative distance, and the cumulative distance calculation adopts Euclidean distance or Manhattan distance.

[0022] Optionally, the method further includes a feedback adaptive mechanism, which monitors the time domain envelope of the enhanced speech signal in real time, compares it with the target time domain envelope again, and continuously updates the local gain correction parameters based on the new difference.

[0023] The advantage of the present invention over the prior art is that it proposes a personalized speech noise reduction and enhancement method based on user-specific time domain envelope reconstruction, which effectively solves the problems of speech distortion and lack of personalization in existing speech noise reduction methods. By automatically collecting personalized speech samples when the user is in a low-noise environment, and performing phoneme-level segmentation and time domain envelope feature extraction on the samples, a user-specific target time domain envelope model is established. When used in real time, the system extracts the time domain envelope of the speech signal in real time, and uses a dynamic time warping algorithm to accurately compare the real-time time domain envelope with the user-specific target envelope to determine accurate local gain correction parameters and realize personalized dynamic gain adjustment. In addition, the present invention further introduces time-domain filtering processing to effectively remove high-frequency spikes and low-frequency drift interference in the speech signal; uses the attention mechanism to optimize the local gain correction parameters to enable more effective correction of phonemes with large differences; adds a frequency-domain processing channel, uses short-time Fourier transform and Hamming window function to effectively remove environmental noise in the spectrum, and integrates it with the time-domain results to improve the quality of speech enhancement; also improves the system's adaptability to different users by normalizing the target time-domain envelope data; and adopts a feedback adaptive mechanism to continuously optimize the enhancement effect to ensure the high quality and stability of speech processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 This is a flow chart of speech sample collection and target envelope model establishment in a low-noise environment of the present invention;

[0025] Figure 2 This is the flow chart of real-time speech signal processing, phoneme segmentation and DTW alignment of the present invention

[0026] Figure 3 This is a flow chart of the dynamic gain adjustment and time domain filtering process of the present invention;

[0027] Figure 4 It is a flow chart of frequency domain processing and feedback adaptive optimization of the present invention. DETAILED DESCRIPTION

[0028] The specific embodiments of the present invention will be described below with reference to the accompanying drawings.

[0029] This invention provides a personalized speech noise reduction and enhancement method based on user-specific time-domain envelope reconstruction. This method addresses the problems of existing speech noise reduction methods, such as speech distortion, reduced sound quality, and a lack of personalization. In individual scenarios, such as headset calls or voice recording, where each user's pronunciation characteristics and speech dynamic range vary significantly, it is difficult to achieve both effective noise reduction and natural speech using a unified noise reduction standard.

[0030] Therefore, if Figure 1 As shown, the present invention first monitors the ambient noise level in real time during a user's daily calls or voice input. When the headset detects that the ambient noise level is below a preset noise threshold (e.g., 50dB or lower), the user's voice signal is clear and largely interference-free. The system automatically captures the voice clip at this time to form a low-noise environment voice sample, ensuring that the collected sample is of sufficient quality to accurately reflect the user's true voice characteristics.

[0031] The present invention then employs a fine-grained, phoneme-level analysis method. Specifically, using a pre-defined phoneme segmentation algorithm (such as a hidden Markov model or a deep learning-based phoneme recognition model), the collected low-noise speech sample is accurately segmented into several phoneme units. For each phoneme unit, its time-domain envelope features are further extracted. These time-domain envelopes are used as baseline features for user-specific speech and stored in a database, forming a user-specific target time-domain envelope model.

[0032] In actual use, the system first performs phoneme-level segmentation on the real-time input voice signal so that each phoneme unit can be processed in detail. The real-time time domain envelope extracted from each phoneme unit uses the following formula:

[0033]

[0034] Where E(n) represents the short-time energy of the real-time speech signal at the nth moment, x(m) represents the sampling value of the real-time speech signal at the mth moment, and L represents the length of the sliding window. For example, the window length can be set to 20 milliseconds to 40 milliseconds to cover the typical short-time dynamic characteristics of human pronunciation.

[0035] like Figure 2As shown, after extracting the real-time time envelope, the present invention uses a dynamic time warping (DTW) algorithm to align the current real-time time envelope with the target time envelope stored in the database on the time axis in order to accurately compare the difference between the current real-time speech and the user's personalized baseline features. After determining the optimal time alignment path through the DTW algorithm, the difference between the real-time time envelope and the target time envelope within each phoneme unit is further calculated to obtain the local gain correction parameters for each phoneme unit in the real-time speech signal.

[0036] like Figure 3 As shown, after completing the above-mentioned difference analysis and parameter calculation, the present invention further implements dynamic gain adjustment to effectively enhance the user's voice and reduce distortion. In specific implementation, firstly, a time domain filter (such as a Butterworth filter or similar low-pass or high-pass filter) is used to process the real-time voice signal to eliminate high-frequency spikes (such as sudden impulse noise) and low-frequency drift (such as fluctuations caused by low-frequency background noise) that appear in the real-time signal, thereby improving the stability and smoothness of the voice signal. Subsequently, the amplitude of the voice signal that has completed time domain filtering is dynamically adjusted phoneme by phoneme unit according to the local gain correction parameters obtained above. Through this two-step processing, while ensuring that the background noise is effectively suppressed, the original dynamic characteristics and personalized pronunciation style of the user's voice can be restored more accurately, significantly improving the voice quality and naturalness.

[0037] In some embodiments, the difference between the real-time time envelope and the target time envelope is calculated using the following formula:

[0038] D(E)=i t (i)―E r (i);

[0039] Where D(i) represents the difference between the two time domain envelopes at the i-th moment, E t (i) represents the specific value of the target time domain envelope at the i-th moment, and E r (i) is the measured time-domain envelope value of the real-time speech signal at the same time. This formula intuitively reflects the gap between the current real-time speech signal and the user's standard speech characteristics, making subsequent parameter adjustments more targeted.

[0040] In some embodiments, an attention mechanism may be introduced in the process of determining the local gain correction parameters. Specifically, the motivation for introducing this mechanism is that the importance of different time periods of the speech signal and the degree of impact on the auditory perception are often different. For example, some phonemes or time periods have small differences and have limited impact on speech quality, while other phonemes or time periods have large differences and will significantly affect the user's auditory experience. The attention mechanism can dynamically identify and highlight those time periods where the real-time time domain envelope differs significantly from the target envelope. By assigning greater weights to these key time periods, the calculation process of the local gain correction parameters is optimized, making the overall speech noise reduction and enhancement effects more balanced and natural.

[0041] In a more specific embodiment:

[0042] The attention weight w(i) is introduced to give higher weight to the period with larger difference. After combining the difference and the attention weight, the local gain correction parameter is defined:

[0043]

[0044] Among them, α represents the adjustable global scaling coefficient, which is used to control the strength of the overall gain correction. w(i) is the attention weight, which can be adaptively selected according to the difference size or phoneme importance. ε is a small constant to prevent the denominator from having a zero value.

[0045] Next, if the amplitude of the speech signal sampling value x(i) needs to be adjusted, it can be multiplied by the corresponding gain coefficient G(i), that is,

[0046] x′(i)=G(i)·x(i);

[0047] This approach can suppress background noise while keeping the enhanced speech closer to the target time-domain envelope, reducing distortion caused by noise reduction. If frequency-domain processing or feedback adaptation mechanisms are combined, α or w(i) can be dynamically adjusted based on the current spectrum state or historical correction results, thereby improving the speech enhancement system's adaptability in complex environments.

[0048] In some embodiments, in order to further improve the noise reduction effect and the accuracy of speech enhancement, the present invention also adds a frequency domain processing channel as an auxiliary optimization approach. Figure 4 As shown, the frequency domain channel uses the Short-Time Fourier Transform (STFT) method to transform the real-time speech signal from the time domain to the frequency domain for analysis and processing. In specific implementations, the STFT uses a Hamming window function, with a window length preferably set between 20 and 40 milliseconds and a frame shift of half the window length. This setting effectively strikes a balance between time resolution and frequency resolution.

[0049] The frequency domain processing channel meticulously removes ambient noise from the speech signal's spectrum, while preserving the spectral components critical to speech perception. The noise reduction results from this frequency domain channel are then effectively integrated with the enhancement results obtained using time domain methods, further improving the clarity, naturalness, and overall listening experience of the final output speech signal.

[0050] By combining the time domain and frequency domain processing channels, the present invention fully utilizes the respective advantages of the two processing methods, and can better achieve high-quality noise reduction and enhancement of speech, especially showing significant advantages and practical value in personalized speech processing.

[0051] The reason for using the Hamming window is its good spectral characteristics, which can effectively reduce the sidelobe effects generated during the frequency domain transformation process, alleviate the problem of speech signal spectrum leakage, and maintain the accuracy of spectrum analysis. At the same time, the present invention sets the window length in the range of 20 milliseconds to 40 milliseconds. The window length in this range is selected based on the actual balance between time and frequency resolution of human speech signals. For example, a shorter window length can improve the time domain resolution and is more suitable for capturing short-term and rapidly changing speech segments, while a longer window length can improve the frequency resolution and is more suitable for processing continuous or low-frequency changing speech components. The present invention sets the frame shift to half the window length. This 50% overlapping window sliding method ensures sufficient overlap and coverage of information between consecutive windows, further enhancing the stability and continuity of spectrum processing.

[0052] In some embodiments, to improve the robustness of user target time envelope data storage and subsequent calculations, the present invention employs a normalization scheme after target time envelope feature extraction. This normalization process specifically adjusts all target time envelope data to between 0 and 1, using the maximum value of the target time envelope as a benchmark. This is primarily intended to eliminate the effects of signal amplitude variations caused by different users, environments, or recording conditions, ensuring greater versatility and consistency in the target time envelope features stored in the database, thereby facilitating accurate and stable comparison and analysis of subsequent real-time speech signals against the target envelope features.

[0053] In some embodiments, during the implementation of the dynamic time warping algorithm, the present invention uses the minimum cumulative distance as the principle to determine the optimal alignment path between the real-time time domain envelope and the target time domain envelope. Specifically, when calculating the cumulative distance, the algorithm can select Euclidean distance or Manhattan distance as a metric. Euclidean distance is suitable for measuring the overall difference in continuous changes between speech signal envelopes, and can better reflect the impact of amplitude differences on the overall envelope matching effect; while Manhattan distance is more suitable for emphasizing the cumulative effect of instantaneous amplitude differences on the time domain envelope, and is suitable for application scenarios that require fine alignment. During implementation, the appropriate distance calculation method can be flexibly selected according to the actual speech processing scenario, signal characteristics and user personalized needs to achieve the best speech matching effect.

[0054] In some embodiments, the present invention specifically introduces a feedback adaptive mechanism to dynamically and in real time optimize and improve the speech noise reduction and enhancement effect. Specifically, the feedback adaptive mechanism monitors the time domain envelope of the speech signal after the above-mentioned noise reduction and enhancement processing in real time, and then compares and analyzes the enhanced real-time speech signal with the pre-stored target time domain envelope one by one to continuously update the local gain correction parameters. The motivation for adopting this feedback loop optimization is that speech signals and environmental noise often have obvious non-steady-state characteristics, and a single fixed parameter or a one-time parameter determination method is often difficult to adapt to the actual complex dynamic changing environment. By adjusting the local gain correction parameters in real time through the feedback mechanism, it is possible to more effectively adapt to the dynamic changes of the actual environment and the user's voice signal, and continuously maintain the high-quality output effect of speech enhancement. This mechanism is particularly suitable for scenarios where the user's voice environment changes over a long period of time, such as applications under unstable conditions such as outdoor or mobile calls and recordings.

[0055] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field, within the technical scope disclosed by the present invention, who makes equivalent replacements or changes based on the technical solution and inventive concept of the present invention, should be covered by the scope of protection of the present invention.

Claims

1. A personalized speech noise reduction and enhancement method based on user-specific time domain envelope reconstruction, characterized in that: The following steps are involved: When the user uses the headset for daily calls or voice input, the ambient noise level is monitored in real time. When the ambient noise level is lower than the preset noise threshold, the user's voice signal at this time is automatically collected as a low-noise environment voice sample; Performing phoneme-level segmentation on the low-noise environment speech sample, extracting the time domain envelope of each phoneme unit, and storing them as target time domain envelopes of the corresponding phonemes in a database; When used in real time, the real-time speech signal is segmented at the phoneme level, and the real-time time domain envelope of each phoneme unit is extracted. After the real-time time domain envelope is time-aligned with the target time domain envelope of the corresponding phoneme in the database using the dynamic time warping algorithm, the difference between the real-time time domain envelope of each phoneme unit and the corresponding target time domain envelope is calculated. Based on the difference, the local gain correction parameters of each phoneme unit in the real-time speech signal are generated; Dynamic gain adjustment is performed on each phoneme unit of the real-time speech signal according to the local gain correction parameter to obtain an enhanced real-time speech signal.

2. The personalized speech noise reduction and enhancement method according to claim 1, characterized in that: The dynamic gain adjustment includes: first performing time domain filtering on the real-time speech signal to eliminate high-frequency peaks and low-frequency drifts in the real-time speech signal; and then using the local gain correction parameter to adjust the amplitude of the real-time speech signal after time domain filtering.

3. The personalized speech noise reduction and enhancement method according to claim 1, characterized in that: The real-time time domain envelope is extracted using the following formula: Where E(n) is the short-time energy of the real-time speech at the nth moment, x(m) is the sampling value of the real-time speech signal, and L is the sliding window length.

4. The personalized speech noise reduction and enhancement method according to claim 1, characterized in that: The difference calculation formula between the real-time time envelope and the user target time envelope is: D(E)=i t (i)―E r (i); Where D(i) is the envelope difference value at the i-th moment, E t (i) is the value of the target time domain envelope at the i-th moment, E r (i) is the value of the real-time time domain envelope at the i-th moment.

5. The personalized speech noise reduction and enhancement method according to claim 1, characterized in that: The local gain correction parameter is determined by introducing an attention mechanism, including using the attention mechanism to give a greater weight to the time period where the real-time time domain envelope is more different from the target time domain envelope, so as to optimize the calculation of the local gain correction parameter.

6. The personalized speech noise reduction and enhancement method according to claim 1, characterized in that: The method also includes a frequency domain processing channel, which uses short-time Fourier transform to perform spectrum analysis on the real-time voice signal and remove environmental noise in the spectrum. After the noise reduction result of the frequency domain channel is integrated with the enhancement result of the time domain channel, the final enhanced real-time voice signal is output.

7. The personalized speech noise reduction and enhancement method according to claim 6, characterized in that: The window function of the short-time Fourier transform in the frequency domain channel is a Hamming window, and the window length is set in the range of 20 milliseconds to 40 milliseconds.

8. The personalized speech noise reduction and enhancement method according to claim 1, characterized in that: The user target time domain envelope is normalized before storage. The normalization is performed using the maximum value of the target time domain envelope so that the target envelope values are all between 0 and 1.

9. The personalized speech noise reduction and enhancement method according to claim 1, characterized in that: The dynamic time warping algorithm determines the optimal alignment path between the real-time time domain envelope and the target time domain envelope based on the principle of minimum cumulative distance, and the cumulative distance calculation adopts Euclidean distance or Manhattan distance.

10. The personalized speech noise reduction and enhancement method according to claim 1, characterized in that: The method further includes a feedback adaptive mechanism, which monitors the time domain envelope of the enhanced speech signal in real time, compares it with the target time domain envelope again, and continuously updates the local gain correction parameter according to the new difference.

Images