An environment-aware controllable background removal and preservation speech synthesis system

Abstract

This invention discloses an environment-aware, controllable background removal and preservation speech synthesis system, relating to the field of speech. The system proposes a speech synthesis system capable of perceiving the acoustic environment based on noisy cues, thereby performing controllable background removal and preservation. It takes text, cues, and task-related control signals as input and includes a duration predictor, an acoustic model, and a dual-cue speech encoder. In terms of training strategy, based on a stream matching algorithm, a controllable masked speech prediction training strategy is further proposed, achieving controllable background removal and preservation by providing noisy cues. This invention improves the robustness and controllability of the system in handling noisy, reverberant, and speaker-interference cues, effectively controlling the removal and preservation of background contained in the cues during speech generation, achieving higher generated speech quality and a more similar acoustic background.

Description

Technical Field

[0001] This invention relates to the field of speech, and more particularly to a speech synthesis system with controllable background removal and preservation based on environmental awareness. Background Technology

[0002] Text-to-speech (TTS) is an assistive technology that converts written text into spoken language. Zero-shot text-to-speech (ZSTTS) aims to generate natural speech from an unseen speaker, conditioned on a prompt. Although advances in deep learning have improved the quality of synthesized speech in recent years, these zero-shot TTS systems still struggle with low-quality prompts or speech contaminated by environmental noise, reverberation, or interfering speakers.

[0003] Acoustic background plays a crucial role in natural speech, providing environmental context and helping listeners understand their surroundings. However, strong background can make it difficult for listeners to hear what is being said. How to handle this acoustic background depends on the specific context: while in some cases background removal is necessary to ensure speech clarity, sometimes preserving the background is essential to maintaining the contextual integrity of the speech.

[0004] There are two main challenges in processing noisy prompts. First, zero-shot text-to-speech systems must remove unwanted background noise from the prompts to generate clean speech with the target speaker's timbre. Second, when acoustic background information in the prompts is important, zero-shot text-to-speech systems must not only mimic the target speaker's timbre but also generate speech with a consistent acoustic background. This invention defines this dual requirement as a background removal and preservation task. The former aims to generate clean speech with a timbre consistent with the target speaker's timbre in the prompts, while the latter requires generating speech with a consistent speaker timbre and a similar acoustic background.

[0005] Most existing research focuses on background removal, aiming to enhance the noise robustness of zero-shot text-to-speech systems to noisy cues. The most direct approach is to apply speech enhancement (SE) techniques to the noisy cues and then input the enhanced cues into the text-to-speech system. However, even state-of-the-art SE models inevitably introduce processing artifacts, reducing the quality of the speech generated by the text-to-speech system. Current research primarily focuses on enhancing the robustness of text-to-speech models in generating clean speech from noisy cues. Recently, research has improved the noise robustness of stream matching models using a masked speech denoising (MSD) strategy. This strategy, during acoustic model training, considers the context of the cues x. ctx Noise is added to enhance the synthesized speech, and the original clean speech target is predicted. However, while this approach is effective in scenarios involving additional noise, it loses the background preservation capability inherent in traditional stream matching models, limiting its application in scenarios where preserving the acoustic environment of the cue is crucial. Furthermore, existing systems reduce the timbre similarity between the synthesized speech and the cue speech compared to the cue speech, especially when dealing with cue speech containing reverberation or interfering speaker noise.

[0006] In contrast, less attention has been paid to the task of background preservation. Some existing systems, such as VoiceLDM and Audiobox, control the speech synthesis of acoustic background through text descriptions, but many acoustic backgrounds cannot be accurately and specifically described in language. EATTS achieves background preservation for reverberant cues, but this system is only effective in scenarios with relatively stable acoustic features such as reverberation. Recent zero-shot text-to-speech models, such as VoiceBox and VALL-E, utilize stream matching models and language models, respectively, and have initially achieved the ability to preserve background, demonstrating the potential of generated speech to reflect the acoustic environment of the input cues. However, systems like Voicebox and VALL-E lack control mechanisms, resulting in unstable speech synthesis results under different cues. In the VoiceBox system, the timbre and background of the cues are intertwined, causing the acoustic background in the generated speech to exist only when there are speech segments, disappearing in silent sections. Furthermore, these existing systems cannot simultaneously remove or preserve environmental information, because background removal and preservation tasks are inherently conflicting and may affect each other's effectiveness.

[0007] Therefore, those skilled in the art are dedicated to developing a system that can simultaneously remove or retain environmental information. Summary of the Invention

[0008] In view of the above-mentioned deficiencies of the prior art, the technical problem to be solved by the present invention is how to develop a system that can simultaneously remove or retain environmental information.

[0009] To achieve the above objectives, this invention provides an environment-aware, controllable background removal and preservation speech synthesis system. The system is characterized by its ability to perceive the acoustic environment based on noisy cues, thereby enabling controllable background removal and preservation. It takes text, cues, and task-related control signals as input and includes a duration predictor, an acoustic model, and a dual-cue speech encoder. In terms of training strategy, based on a stream matching algorithm, a controllable masked speech prediction training strategy is further proposed, achieving controllable background removal and preservation by providing noisy cues.

[0010] Furthermore, the text is used to control the content of the synthesized speech, the prompting speech is used to control the speaker's timbre and acoustic background of the synthesized speech, and the task-related control signal is used to control the synthesized speech to be clean speech (or the synthesized speech to have an acoustic background similar to the prompting speech).

[0011] Furthermore, the acoustic model models the conditional distribution qxz,x ctx Based on the phoneme sequence z and the cue x ctx A Mel spectrogram x is generated, and the acoustic model, acting as a vector field estimator, predicts a randomly selected mask portion at each timestamp t∈0,1. At the same time, the visible part x ctx =x⊙1mask is considered a prompt; the training objective of the acoustic model is As described, the output of network θ is v t w,x ctx The flow of steps z; θ, t is denoted as w, while x0 is Gaussian noise sampled from a normal distribution, σ min It is a hyperparameter of the control flow matching deviation.

[0012] Furthermore, both the duration predictor and the acoustic model employ a classifier-free guided strategy (CFG) during training and testing to balance pattern coverage and sample fidelity; during the training of the acoustic model, acoustic cue x ctx The phoneme sequence z with probability p uncond Randomly discarded; during inference, the acoustic model first samples a Gaussian noise x0 from the normal distribution p0 and uses the ODE solver to evaluate the flow w; the modified vector field under the CFG strategy becomes the formula In Instead of v t w,x ctx,z;θ, where α is a hyperparameter controlling the guiding intensity.

[0013] Furthermore, the duration predictor models the conditional distribution qyl,y ctx , where y is the predicted duration sequence for each phoneme, given the input phoneme index sequence l and the cue y. ctx Therefore, the training objective and inference process of the duration predictor are respectively formulated as follows: and formula

[0014]

[0015] Furthermore, the acoustic model proposes a novel controllable masked speech prediction strategy, which unifies the background removal and preservation tasks into a mask prediction problem. By introducing task-related control signals, the system is guided to accurately switch between background removal and preservation, achieving dual-objective control.

[0016] Furthermore, in the controllable masked speech prediction strategy, the acoustic model uses probability P N P R and P IS Modeling different types of noise involves simulating enhanced noisy speech x by adding ambient noise, reverberation, and interfering speech to clean training samples x, respectively. aug The probability of not adding a background, i.e., x aug =x, denoted as P C A portion of the speech is randomly masked and needs to be predicted, denoted as before and after enhancement, respectively. and The unmasked portion is used as a hint, denoted as A multi-task learning loss was designed based on the conventional stream matching loss, as shown in the formula:

[0017]

[0018] Among them, acoustic model The output is v t w,x ctx The flow of steps z, c; θ, t is denoted as w, x0 is Gaussian noise sampled from a normal distribution, and σ min It is a hyperparameter of the control flow matching deviation.

[0019] Furthermore, two identical encoders are introduced into the acoustic model, which independently process the noisy prompts.

[0020] Furthermore, a control mechanism guided by binary control signal c determines which speaker encoder branch to activate; this architecture design minimizes model interference between removal and retention tasks while further improving background retention performance.

[0021] Furthermore, during the inference process, given a control signal c for a specified task, the inference process first involves the duration predictor generating a phoneme sequence z based on a given transcription; then, noise cue x... ctx The phoneme sequence z, Gaussian noise x0, and control signal c are input into the acoustic model. A modified CFG strategy is employed, and the acoustic model, guided by an ordinary differential equation solver and a defined vector field, predicts the final Mel spectrum, as shown in the formula. Finally, the HiFiGAN vocoder converts the Mel spectrogram into a waveform.

[0022] The present invention has the following technical effects:

[0023] Through a dual background processing mechanism, the system's robustness in noisy environments is significantly enhanced. Compared to traditional systems, the background removal module of this invention can more accurately filter out unwanted background noise, ensuring that the output speech maintains high quality under various background noise conditions. Preliminary experimental data show that this system significantly improves speech clarity and naturalness in different noise scenarios, especially in scenarios with complex acoustic environments.

[0024] For scenarios requiring consistent acoustic background in prompts, this invention achieves high standards in both timbre matching and background consistency, further enhancing the realism and immediacy of the generated speech. Compared to other methods, this invention outperforms existing technologies in timbre similarity metrics (such as timbre matching score) and background consistency scores in background preservation tasks, demonstrating its industrial application value.

[0025] The following will further explain the concept, specific structure, and technical effects of the present invention in conjunction with the accompanying drawings, so as to fully understand the purpose, features, and effects of the present invention. Attached Figure Description

[0026] Figure 1 This is a system overview diagram of a preferred embodiment of the present invention. Detailed Implementation

[0027] The following description, with reference to the accompanying drawings, illustrates several preferred embodiments of the present invention to make its technical content clearer and easier to understand. The present invention can be embodied in many different forms, and the scope of protection of the present invention is not limited to the embodiments mentioned herein.

[0028] In the accompanying drawings, components with the same structure are indicated by the same numerical designation, and components with similar structures or functions are indicated by similar numerical designations. The dimensions and thicknesses of each component shown in the drawings are arbitrary, and the present invention does not limit the dimensions and thicknesses of each component. To make the illustrations clearer, the thickness of some components has been appropriately exaggerated in the drawings.

[0029] This invention proposes a speech synthesis system capable of perceiving the acoustic environment based on noisy cue speech, thereby performing controllable background removal and preservation. The system takes text, cue speech, and task-related control signals as inputs. The text controls the content of the synthesized speech, the cue speech controls the speaker's timbre and acoustic background, and the task-related control signals control whether the synthesized speech is clean (without acoustic background) or has an acoustic background similar to the cue speech. The entire system is structurally as follows: Figure 1 It includes a duration predictor, an acoustic model, and a dual-prompt speech encoder. Regarding the training strategy, this invention, based on a stream matching algorithm, further proposes a controllable masked speech prediction training strategy, achieving controllable background removal and preservation by providing noisy prompts.

[0030] I. Duration Predictor and Acoustic Model Based on Flow Matching

[0031] Flow matching (FM) is a simulation-free method for training CNF. The relationship between the vector field and the flow is defined by ordinary differential equations (ODEs). The text-to-speech system of this invention consists of an acoustic model based on flow matching and a duration predictor.

[0032] The acoustic model models the conditional distribution qxz,x ctx Specifically, it is based on the phoneme sequence z and the cue x. ctx Generate the Mel spectrogram x. This model acts as a vector field estimator, predicting a randomly selected mask portion at each timet t∈0,1. At the same time, the visible part x ctx =x⊙1-mask is considered a cue. The training objective of the acoustic model is as shown in the formula. As described, the output of network θ is v t w,x ctx The flow at steps z, θ, and t is denoted as w, while x0 is Gaussian noise sampled from a normal distribution. σ min It is a hyperparameter of the control flow matching deviation.

[0033] Both the duration predictor and the acoustic model employ a classifier-free guidance (CFG) strategy during training and testing to balance pattern coverage and sample fidelity. During the training of the acoustic model, acoustic cues x...ctx The phoneme sequence z with probability p uncond Randomly discarded. During inference, the model first samples a Gaussian noise x0 from the normal distribution p0 and uses the ODE solver to evaluate the flow w. The modified vector field under the CFG policy becomes the formula. In Instead of v t w,x ctx ,z;θ, where α is a hyperparameter controlling the guiding intensity.

[0034] The duration predictor is similar to the acoustic model, which models the conditional distribution qyl,y ctx , where y is the predicted duration sequence for each phoneme, given the input phoneme index sequence l and the cue y. ctx Therefore, the training objective and inference process of the duration predictor are respectively formulated as follows: and formula

[0035]

[0036] II. Controllable Masked Speech Prediction Training Strategy

[0037] While masked speech denoising strategies effectively estimate clean audio from noisy cues, they cannot selectively preserve the acoustic environment. To address this issue, this invention proposes a novel Controllable Masked Speech Prediction (CMSP) strategy for acoustic models, unifying background removal and preservation tasks into a mask prediction problem. This enables the model to perform dual mask predictions within the same training framework, ensuring the model can flexibly adapt to different application scenarios while maintaining consistent speaker characteristics.

[0038] Specifically, through probability P N P R and P IS To model different types of noise, this invention simulates enhanced noisy speech x by adding environmental noise, reverberation, and interference speech to clean training samples x, respectively. aug The probability of not adding a background, i.e., x aug =x, denoted as P C A portion of the speech is randomly masked and needs to be predicted, denoted as before and after enhancement, respectively. and The unmasked portion is used as a hint, denoted as This invention designs a multi-task learning loss based on conventional flow matching loss, as shown in Equation 5. The output of the acoustic model θ is v. t w,x ctxThe flow of steps z, c; θ, t is denoted as w, while x0 is Gaussian noise sampled from a normal distribution. σ min This is the hyperparameter of the control flow matching deviation. Other parameters remain consistent with Equation 1.

[0039] By introducing binary control signals cc0 and c1, the model is guided to focus on the removal (corresponding to control signal c0) or retention (corresponding to control signal c1) of background elements, thereby achieving precise control over the acoustic environment of the generated speech.

[0040]

[0041] III. Dual-prompt voice encoder

[0042] Previous research has shown that it is crucial to preserve the speaker's timbre in synthesized speech when processing noisy cues. However, other systems, such as VoiceBox, which simply concatenate noisy cues into the model input, may limit their ability to accurately extract speaker information under complex acoustic conditions. To address this limitation, this invention proposes a dual-cue speech encoder specifically designed for better handling of noisy cues. Figure 1 As shown, in addition to concatenating the noisy cues into the input, this invention introduces two identical encoders into the model. These encoders independently process the noisy cues, and a control mechanism guided by a binary control signal 'c' determines which speaker encoder branch is activated. This architecture minimizes model interference between the removal and retention tasks while further improving background preservation performance.

[0043] During inference, given a control signal c for a specific task, the inference process first involves a duration predictor generating a phoneme sequence z based on a given transcription. Then, a noise cue x... ctx The phoneme sequence z, Gaussian noise x0, and control signal c are input into the acoustic model. Correspondingly, this invention employs a modified CFG strategy. The acoustic model, guided by an ODE solver and a defined vector field, predicts the final Mel spectrum, as shown in the formula. As shown. Finally, the HiFiGAN vocoder converts the Mel spectrogram into a waveform.

[0044] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A speech synthesis system with controllable background removal and preservation based on environmental awareness, characterized in that, This invention proposes a speech synthesis system capable of perceiving the acoustic environment based on noisy prompts, thereby enabling controllable background removal and preservation. The system takes text, prompts, and task-related control signals as input and includes a duration predictor, an acoustic model, and a dual-prompt speech encoder. In terms of training strategy, based on the stream matching algorithm, a controllable masked speech prediction training strategy is further proposed, which achieves controllable background removal and preservation by providing noisy prompts. The text is used to control the content of the synthesized speech, the prompt speech is used to control the speaker's timbre and acoustic background of the synthesized speech, and the task-related control signal is used to control the synthesized speech to be clean speech or to have an acoustic background similar to the prompt speech, wherein clean speech is speech without acoustic background. The acoustic model models the conditional distribution. According to the phoneme sequence and prompts Generate Mel spectrum The acoustic model serves as a vector field estimator at each time stamp. Predict a randomly selected mask ( )part At the same time, the visible part This is considered a prompt; the training objective of the acoustic model is... The network described The output is , The flow of steps is denoted as ,and It is Gaussian noise sampled from a normal distribution. It is a hyperparameter of the control flow matching deviation; Both the duration predictor and the acoustic model employ a classifier-free guided strategy (CFG) during training and testing to balance pattern coverage and sample fidelity; during the training of the acoustic model, acoustic cues are provided. phoneme sequence With probability Randomly discarded; during the inference process, the acoustic model first starts from a normal distribution. Sampling a Gaussian noise And use the ODE solver to evaluate the flow. The modified vector field under the CFG strategy becomes the formula. ,in It is a hyperparameter that controls the guiding strength; The duration predictor models a conditional distribution. ,in It is the predicted duration sequence for each phoneme, given the input phoneme index sequence. and prompts Therefore, the training objective and inference process of the duration predictor are respectively formulated as follows: and formula ; The acoustic model proposes a novel controllable masked speech prediction strategy, which unifies the background removal and preservation tasks into a mask prediction problem. By introducing task-related control signals, the system is guided to switch precisely between background removal and preservation, achieving dual-objective control.

2. The speech synthesis system with controllable background removal and preservation based on environmental awareness as described in claim 1, characterized in that, In the controllable masked speech prediction strategy, the acoustic model uses probability... , and Modeling different types of noise is achieved by adding ambient noise, reverberation, and interfering speech to clean training samples, respectively. In this way, we can simulate the enhanced noisy speech. The probability of not adding a background, i.e. , recorded as A portion of the speech is randomly masked and needs to be predicted, denoted as before and after enhancement, respectively. and The unmasked portion serves as a hint, denoted as A multi-task learning loss was designed based on the conventional flow matching loss, as shown in the formula: Among them, acoustic model The output is , The flow of steps is denoted as , It is Gaussian noise sampled from a normal distribution. is the hyperparameter of the control flow matching bias, and c is the control signal, including the removal signal c0 and the retention signal c1 that guide the model to focus on background elements.

3. The speech synthesis system with controllable background removal and preservation based on environmental awareness as described in claim 2, characterized in that, Two identical encoders are introduced into the acoustic model, corresponding to the background cancellation encoder and the background preservation encoder, which process the noisy prompt speech independently.

4. The speech synthesis system with controllable background removal and preservation based on environmental awareness as described in claim 3, characterized in that, Through binary control signals The bootstrap control mechanism determines which encoder branch to activate.

5. The speech synthesis system with controllable background removal and preservation based on environmental awareness as described in claim 4, characterized in that, During the reasoning process, control signals for a given task are provided. The reasoning process begins with the duration predictor generating a phoneme sequence based on a given transcription. Then, noise alert Phoneme sequence Gaussian noise and control signals This information will be input into the acoustic model; a modified CFG strategy is employed, and the acoustic model, guided by an ordinary differential equation solver and a defined vector field, predicts the final Mel spectrum, as shown in the formula. Finally, the HiFiGAN vocoder converts the Mel spectrogram into a waveform.

Images