A general-purpose speech enhancement system with low hallucination characteristics
By constructing a general speech enhancement framework consisting of a packet loss detection unit, a basic speech enhancement model, an adapter, and a vocoder, the speech consistency and perception quality issues of existing systems in multi-distortion scenarios are solved, achieving high-fidelity, low-illusion speech recovery and supporting multiple sampling rate inputs and outputs.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- NANJING UNIV
- Filing Date
- 2026-03-26
- Publication Date
- 2026-07-10
AI Technical Summary
Existing general-purpose speech enhancement systems suffer from insufficient subjective perceived quality, inadequate speech consistency, and limited adaptability to multi-distortion scenarios. They perform particularly poorly under complex distortion conditions such as bandwidth constraints and packet loss. Furthermore, self-supervised speech models are insufficient in maintaining consistency between speech content and speaker features.
A unified general speech enhancement framework is constructed, including a packet loss detection unit, a speech enhancement base model, an adapter, and a vocoder. Through packet loss detection, self-supervised pre-training, phoneme representation extraction, and acoustic mapping, high-fidelity speech recovery is achieved, and multiple sampling rate inputs and outputs are supported.
It can recover high-quality speech under various distortion conditions, maintain the consistency of speech content and speaker characteristics, and has robustness and universality, thus improving the perceptual quality and reliability of speech enhancement.
Smart Images
Figure CN122369480A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of speech enhancement, and specifically relates to a general speech enhancement system with low illusion characteristics. Background Technology
[0002] Universal Speech Enhancement (USE) aims to recover speech signals affected by various distortions and is applicable to different input formats, such as different sampling rates and number of channels. In recent years, to promote the development of universal speech enhancement technology and establish a unified evaluation standard, the URGENT (Universality, Robustness, and Generalizability of Speech Enhancement) challenge has been held, providing a unified evaluation benchmark for related technologies. In URGENT 2025, the task required building a single model capable of handling seven types of speech distortion, including noise, reverberation, clipping, bandwidth limitation, coding distortion, packet loss, and wind noise, and supporting multiple sampling rates (8, 16, 22.05, 24, 32, 44.1, 48kHz).
[0003] Existing general-purpose speech enhancement systems have made some progress in multi-distortion processing and multi-sampling rate adaptation, but several technical problems still exist:
[0004] (1) Insufficient subjective perception quality: Although some existing methods perform well in terms of objective indicators, they still have limitations in terms of subjective perception quality.
[0005] (2) Insufficient speech consistency: In the process of restoring speech, some methods may result in the speech content deviating from the original speech or the speaker's characteristics being distorted, affecting the authenticity and reliability of the speech.
[0006] (3) Limited adaptability to multiple distortion scenarios: Existing methods usually handle common distortions such as noise, reverberation and clipping well, but their performance drops significantly for more challenging distortion types such as bandwidth-limited and packet loss.
[0007] Furthermore, self-supervised speech models (S3M) are widely used in speech enhancement tasks, providing valuable information for speech recovery by extracting high-level features or phoneme-level representations of speech. However, existing methods still have shortcomings in fully utilizing the structured information of these representations, maintaining the consistency of speech content, and preserving speaker features.
[0008] Therefore, there is an urgent need for a general speech enhancement method that can improve the perceptual quality of enhanced speech while maintaining consistency between speech content and speaker features, and maintain good robustness and reliability in multi-distortion scenarios. Summary of the Invention
[0009] This invention proposes a universal speech enhancement system with low-illusion characteristics, capable of recovering high-quality speech under various speech distortion conditions while maintaining consistency in speech content and speaker features. To achieve this goal, this invention constructs a unified universal speech enhancement framework, including a packet loss detection unit, a basic speech enhancement model, an adapter, a vocoder, and a post-processing network.
[0010] Specifically, the technical solution of the present invention includes the following:
[0011] Packet Loss Detection Unit: The packet loss detection unit is used to detect the packet loss location of the input speech signal and generate mask information indicating the time and location of the packet loss. This mask information can guide the speech enhancement base model to perform context-based phoneme prediction at the location of the lost segment, thereby enhancing the ability to recover speech from continuous packet loss or long-term missing segments.
[0012] The speech enhancement base model is obtained by supervised fine-tuning of a self-supervised pre-trained speech model on speech enhancement data. It can extract self-supervised speech representations under noisy or other distortion conditions and, combined with packet loss masking information, perform general speech enhancement operations within a phoneme representation domain constrained by phoneme priors. The model simultaneously outputs enhanced phoneme representations and coarse-grained enhanced acoustic representations. The enhanced phoneme representations provide high-level semantic information, helping to improve the accuracy of speech content; the coarse-grained enhanced acoustic representations contain rich low-level acoustic details, which helps improve speaker feature consistency and the fidelity of speech waveform reconstruction.
[0013] Adapter: The adapter takes the enhanced phoneme representation as the main input and combines it with coarse-grained enhanced acoustic representation as conditional information to generate the enhanced acoustic representation. Through this conditional mapping method, the enhanced acoustic representation combines speech content consistency and acoustic fidelity, thereby further improving the quality of the speech waveform.
[0014] Vocoder: The vocoder is used to convert the enhanced acoustic representation into a speech waveform, thereby enabling the audible output of the speech signal.
[0015] Post-processing network: The post-processing network is used to perform bandwidth expansion processing on the speech waveform.
[0016] The input speech signal undergoes sampling rate adaptation processing before entering the system; and the enhanced speech signal is downsampled to the same sampling rate as the input speech before output, thereby enabling the system to support adaptive processing of multiple sampling rate inputs and outputs.
[0017] The universal speech enhancement system of this invention is applicable to at least one or more types of speech distortion, including noise, reverberation, bandwidth limitation, coding distortion, packet loss distortion, or combinations thereof. Through the synergistic effect of the above modules, this invention achieves the following technical effects:
[0018] (1) It achieves high-fidelity speech recovery under various distortion conditions and has robustness;
[0019] (2) It supports multiple sampling rate inputs and outputs and has versatility.
[0020] In summary, this invention provides a general-purpose speech enhancement system with a complete structure and stable performance, capable of generating high-fidelity, low-illusion enhanced speech under various complex distortion conditions, meeting the requirements of practical applications for speech quality and reliability. Attached Figure Description
[0021] Figure 1 This is a network system architecture diagram of the present invention;
[0022] Figure 2 This is a structural diagram of the basic speech enhancement model in an embodiment of the present invention;
[0023] Figure 3 This is a structural diagram of the adapter in an embodiment of the present invention;
[0024] Figure 4 This is a structural diagram of the vocoder in an embodiment of the present invention;
[0025] Figure 5 This is a diagram of the post-processing network structure in an embodiment of the present invention; Detailed Implementation
[0026] The technical solution of the present invention will be fully and clearly described below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.
[0027] 1. System Overall Structure
[0028] like Figure 1 As shown, the general speech enhancement system of the present invention includes: a packet loss detection unit, a speech enhancement basic model, an adapter, a vocoder, and a post-processing network. The system input is the speech signal to be enhanced at an arbitrary sampling rate, and the output is the enhanced speech signal at the corresponding sampling rate. The system structure and data processing flow are as follows:
[0029] (1) Input speech signal: The original speech signal can be any sampling rate. Before entering the system, the sampling rate is adapted. In this embodiment, it is uniformly set to 16kHz to ensure that the sampling rate of the speech enhancement basic model is consistent.
[0030] (2) Packet Loss Detection Unit: Performs packet loss detection on the sampled rate-adapted speech signal and generates a binary mask information indicating the time location of packet loss. This mask information is used to guide the speech enhancement base model to perform context-based phoneme representation recovery at the packet loss location.
[0031] (3) Basic speech enhancement model: It is obtained by supervised fine-tuning of a self-supervised pre-trained speech model on speech enhancement data, and can extract self-supervised speech representations under noisy or other distortion conditions. After receiving packet loss mask information, the model performs general speech enhancement operations in the phoneme representation domain constrained by phoneme priors, and outputs the enhanced phoneme representation and coarse-grained enhanced acoustic representation.
[0032] In this embodiment, as Figure 2 As shown, the speech enhancement model is based on WavLM and consists of a convolutional encoder and a Transformer encoder. The convolutional encoder has 7 convolutional layers, which encode the input speech signal (dimension B×L) into frame-level features (dimension B×D×T). The corresponding packet loss frame position is covered by a learnable mask vector and then input into the Transformer encoder for further processing, finally outputting the enhanced speech representation (dimension B×D×T).
[0033] (4) Adapter: such as Figure 3 As shown, an improved Vocos architecture is adopted, including a front-end of two ResNet layers, a front-end of one Attention layer, a rear-end of two ResNet layers, and a 12-layer ConvNeXt layer. The enhanced phoneme representation (dimension B×D×T) output by the speech enhancement base model and the coarse-grained enhanced acoustic representation (dimension B×D×T) are concatenated along the channel dimension and input into the adapter to output the enhanced acoustic representation (dimension B×D×T), ensuring speech content consistency and acoustic fidelity.
[0034] (5) Vocoder: such as Figure 4 As shown, the enhanced acoustic representation output by the adapter is converted into a speech waveform. The vocoder employs an improved Vocos architecture and introduces an inverse short-time Fourier transform (iSTFT) module to reconstruct the 16kHz speech waveform, achieving audible output.
[0035] (6) Post-processing network: such as Figure 5As shown, bandwidth expansion processing is performed on the speech waveform generated by the vocoder to generate a full-bandwidth enhanced speech signal. The post-processing network adopts a CWS-TF-GridNet architecture, combining Channel Subband (CWS) and TF-GridNet modules to enhance full-bandwidth spectrum modeling capabilities. To preserve low-frequency content and avoid the impact of bandwidth expansion on low frequencies, the network directly copies the input low-spectral components during the inference phase and applies a transition band between low and high frequencies to ensure smooth spectrum transition. Finally, the output full-spectrum signal is downsampled to generate an enhanced speech signal with the same sampling rate as the input speech.
[0036] 2. System Operation Steps
[0037] The implementation steps of the method of the present invention can be summarized as follows:
[0038] Step S1: Sampling rate adaptation. The input speech signal is subjected to sampling rate adaptation processing to meet the processing requirements of the speech enhancement base model. For example, speech with any sampling rate is resampled to the sampling rate used for model pre-training, such as 16kHz.
[0039] Step S2: Packet loss detection. Packet loss detection is performed on the audio signal after sampling rate adaptation, generating a binary mask. In the mask, "1" indicates packet loss or missing valid frames in the corresponding time period, while "0" indicates that the time period is normal. This mask information is used to guide the speech enhancement base model in predicting and recovering contextual phonemes at the location of lost frames.
[0040] Step S3: Phoneme Representation Extraction and Enhancement. Using a self-supervised pre-trained and supervisedly fine-tuned speech enhancement model, phoneme representations are extracted with the aid of packet loss masking information. General speech enhancement operations are then performed within the phoneme representation domain, constrained by phoneme priors, thereby generating enhanced phoneme representations and coarse-grained enhanced acoustic representations. The phoneme priors can be obtained by pre-training the self-supervised pre-trained speech model on large-scale speech data through mask prediction, used to limit the generated content from deviating from the input speech, thus reducing hallucinations. The coarse-grained enhanced acoustic representations contain richer low-level acoustic details than the phoneme representations, used for subsequent acoustic reconstruction.
[0041] Step S4: Acoustic mapping. Using the enhanced phoneme representation as the primary input and combining it with coarse-grained enhanced acoustic representation as conditional information, the adapter module maps and generates the enhanced acoustic representation. This mapping ensures that the enhanced acoustic representation maintains both speech content consistency and acoustic fidelity.
[0042] Step S5: Waveform reconstruction, input the enhanced acoustic representation into the vocoder and convert it into a speech waveform.
[0043] Step S6: Bandwidth expansion. The post-processing network performs bandwidth expansion processing on the speech waveform to obtain an enhanced speech signal with full bandwidth.
[0044] Step S7: Output sampling rate adjustment, downsampling the bandwidth-extended speech signal to the same sampling rate as the original input speech, completing the final output of the enhanced speech signal, thereby realizing the general enhancement processing of multi-sampling-rate input speech.
[0045] 3. Training methods for basic speech enhancement models
[0046] This invention also provides a method for training a speech enhancement base model, comprising the following steps:
[0047] Step T1: Provide a self-supervised pre-trained speech model, such as WavLM or a variant thereof, which is based on the mask prediction training paradigm.
[0048] Step T2: Obtain speech enhancement training data, which includes data pairs consisting of noisy or otherwise distorted speech and corresponding clean speech.
[0049] Step T3: Based on the training data, perform supervised fine-tuning of the self-supervised pre-trained speech model to form a basic speech enhancement model. A distillation mechanism can be introduced during the fine-tuning process, using the teacher model output as a supervisory signal to improve the model's enhancement performance.
[0050] Through the above implementation methods, the present invention can achieve high-fidelity, low-illusion speech enhancement under various speech distortion conditions, while taking into account the consistency of speech content and speaker feature recovery, and supporting the versatility of multi-sampling rate input and output.
[0051] 4. General voice enhancement performance
[0052] This invention employs standard benchmark datasets in the field of general speech enhancement for training and is validated on independent test sets for multiple subtasks, including: the DNS competition test set for evaluating noise reduction and dereverberation performance, the PLC competition test set for evaluating packet loss compensation capability, and the URGENT test set for evaluating general speech enhancement performance. Test results demonstrate that the general speech enhancement system described in this invention exhibits excellent performance in various speech enhancement subtasks, and also demonstrates superior enhancement effects on comprehensive tasks covering multiple distortion types, verifying the effectiveness and robustness of this technical solution.
Claims
1. A universal speech enhancement system with low hallucination properties, characterized in that, include: The packet loss detection unit is used to detect the packet loss location of the input voice signal and generate mask information indicating the time and location of the packet loss. The speech enhancement base model is obtained by supervised fine-tuning of a self-supervised pre-trained speech model on speech enhancement data. It is used to extract self-supervised speech representations under noisy or other distortion conditions, and combined with the packet loss location mask information, to perform general speech enhancement operations in the phoneme representation domain constrained by phoneme priors, while outputting the enhanced phoneme representation and coarse-grained enhanced acoustic representation. An adapter is used to map and generate an enhanced acoustic representation by taking the enhanced phoneme representation as the main input and combining it with the coarse-grained enhanced acoustic representation as conditional information. A vocoder is used to convert the enhanced acoustic representation into a speech waveform; A post-processing network is used to perform bandwidth expansion processing on the speech waveform to obtain a full-bandwidth enhanced speech signal; The input speech signal undergoes sampling rate adaptation processing before entering the system; and the enhanced speech signal is downsampled to the same sampling rate as the input speech before output, thereby enabling the system to support adaptive processing of multiple sampling rate inputs and outputs.
2. The system as described in claim 1, characterized in that: The speech enhancement base model is obtained by supervised fine-tuning of a self-supervised pre-trained speech model on speech enhancement data, wherein the speech enhancement data includes data pairs consisting of noisy speech or other distorted speech and corresponding clean speech; a distillation mechanism is introduced in the fine-tuning process to use the output of the teacher model as a supervision signal.
3. The system as described in claim 1, characterized in that: The self-supervised pre-trained speech model is WavLM or a variant thereof, which is based on the mask prediction self-supervised training paradigm.
4. The system as described in claim 1, characterized in that: The phoneme prior is derived from the phoneme-level or sub-phoneme-level statistical regularities obtained by the self-supervised pre-trained speech model through mask prediction pre-training on large-scale speech data. This phoneme prior is used to limit the space of speech content that can be generated during generative speech enhancement, thereby reducing speech content illusion.
5. The system as described in claim 1, characterized in that: The general speech enhancement operation is performed in the phoneme representation domain, rather than directly in the time-domain waveform or time-spectrum domain, thereby improving the consistency of speech content and the stability of generation.
6. The system as described in claim 1, characterized in that: The mask information generated by the packet loss detection unit is used to indicate the location of the lost speech segment, so as to guide the speech enhancement basic model to perform context-based phoneme prediction at the corresponding location, thereby enhancing the ability to recover continuous packet loss or long-term missing speech.
7. The system as described in claim 1, characterized in that: The coarse-grained enhanced acoustic representation contains richer low-level acoustic detail information than the enhanced phoneme representation, thereby improving the fidelity of speech waveform reconstruction.
8. The system as described in claim 1, characterized in that: The adapter fuses the enhanced phoneme representation with the coarse-grained enhanced acoustic representation through conditional mapping to generate an enhanced acoustic representation that combines speech content consistency and acoustic fidelity.
9. The system as described in claim 1, characterized in that: The system is suitable for enhancing at least one of the following speech distortion types: noise, reverberation, bandwidth-limited, coding distortion, packet loss distortion, or a combination thereof.
10. A general speech enhancement method with low hallucination properties, characterized in that, Includes the following steps: S1, performs sampling rate adaptation processing on the input speech signal to match the processing sampling rate of the speech enhancement base model; S2, perform packet loss detection on the voice signal after sampling rate adaptation, and obtain mask information indicating the location of packet loss; S3 utilizes a self-supervised pre-trained and supervised fine-tuned speech enhancement base model to extract phoneme representations with the assistance of packet loss masking information, and performs general speech enhancement within the phoneme representation domain constrained by phoneme priors to obtain enhanced phoneme representations and coarse-grained enhanced acoustic representations. S4, based on the enhanced phoneme representation and combined with the coarse-grained enhanced acoustic representation as conditional information, maps and generates the enhanced acoustic representation. S5, convert the enhanced acoustic representation into a speech waveform; S6 performs bandwidth expansion processing on the speech waveform to obtain a full-bandwidth speech signal; S7 downsamples the full-bandwidth speech signal to the same sampling rate as the input speech, and outputs an enhanced speech signal; In step S3, phoneme prior constraints are used to limit the generated language information from deviating from the input speech, thereby reducing the hallucination phenomenon in the speech enhancement process.
11. A training method for a speech enhancement basic model, characterized in that, Includes the following steps: T1 provides a self-supervised pre-trained speech model; T2, acquire speech enhancement training data, the speech enhancement training data including data pairs consisting of noisy or other distorted speech and corresponding clean speech; T3. Based on the speech enhancement training data, the self-supervised pre-trained speech model is subjected to supervised fine-tuning to obtain a speech enhancement base model; wherein, a distillation mechanism is introduced in the supervised fine-tuning process to use the output of the reference model as a supervision signal.