An efficient training and high expressiveness speech conversion model based on an acoustic model and a vocoder decoupled architecture

Abstract

This invention discloses an efficient training and high-performance speech conversion model based on a decoupled architecture of acoustic model and vocoder, comprising an acoustic model and a vocoder; the acoustic model includes a speaker encoder, a content encoder, a normalized stream, a posterior encoder, a Mel decoder, and a discriminator. Its advantages include significant technological breakthroughs in improving speech conversion model training efficiency, sound quality, emotional expression, and interactive control, providing a novel solution for high-quality, highly controllable speech synthesis systems, and possessing good practical value and promising industrial application prospects.

Description

Technical Field

[0001] This invention relates to the field of speech conversion technology, and in particular to an efficient training and high-performance speech conversion model based on an acoustic model and a decoupled architecture of vocoder. Background Technology

[0002] Voice conversion (VC) refers to converting the timbre of input speech into the timbre of another target speaker without altering the content of the speech. The process is as follows: Figure 1 As shown in the image. This technology is widely used in fields such as voice personalization, voice cloning, voice spoofing, voice restoration, and entertainment.

[0003] Voice conversion technology has undergone a significant evolution from traditional statistical modeling to deep generative models. Early methods mainly relied on statistical techniques such as Gaussian Mixture Models (GMMs) to achieve conversion between source and target speech through feature mapping, but faced problems such as low conversion quality and insufficient speech naturalness. With the development of deep learning, nonlinear modeling methods based on neural networks have introduced stronger expressive power and improved conversion results. In recent years, generative adversarial networks, variational autoencoders, stream models, and their combinations have been widely used, greatly enhancing the generative ability and flexibility of models, especially their adaptability to unpaired data. At the same time, the introduction of self-supervised learning and large-scale pre-trained models has made it possible to decouple content from speaker representation.

[0004] Currently, most mainstream speech conversion models employ pre-trained self-supervised models, such as HuberT and WavLM, to extract timbre-independent speech content features from the input speech. These features, to some extent, decouple the speech content from the speaker's information. Subsequently, an acoustic model combining a variational autoencoder and a normalized flow is used to model the content features, generating an intermediate acoustic representation corresponding to the target speaker's timbre. Finally, a vocoder is used to restore this acoustic representation to a speech waveform, achieving the speaker conversion effect. The specific conversion process is as follows: Figure 2As shown. During training, firstly, the content encoder takes the content features of the source speech as input, extracts the speaker-de-identified content features, and constructs its prior distribution; simultaneously, the speaker encoder extracts the speaker vector of the target speaker. Then, the posterior encoder encodes the true linear spectrum to obtain the posterior distribution of the latent variable (z). Conditional on the speaker vector, a normalization flow aligns the prior and posterior distributions and constrains them using KL divergence loss. Next, the latent variable and speaker vector are input into the HiFi-GAN vocoder to generate a speech waveform. Adversarial training is employed using a discriminator, and the entire network iteratively updates the model parameters with KL divergence loss, Mel spectrum reconstruction loss, and speaker reconstruction loss as joint optimization objectives. During inference, firstly, the content encoder extracts pre-trained features from any source speech and generates a content vector, while simultaneously extracting the speaker vector of the target speaker; then, the content features and speaker vector are input into the normalization flow to sample the latent variable, and this latent variable, along with the speaker vector, is fed into the HiFi-GAN vocoder to generate speech with the target speaker's timbre. However, this sound conversion technology has the following problems:

[0005] 1. Low training efficiency: Most mainstream voice conversion models currently employ a joint training approach that integrates the acoustic model and vocoder into the same network. While this approach simplifies the structure, the vocoder typically contains multiple computationally expensive upsampling modules, severely impacting overall training efficiency. Furthermore, this highly coupled structure increases debugging complexity; if model performance becomes abnormal, it's difficult to accurately determine whether the problem originates in the acoustic modeling or the vocoder.

[0006] 2. Insufficient emotional expressiveness: When faced with emotionally rich speech input, existing models often fail to fully reproduce the emotional characteristics of the original speaker, and the emotional details of the converted speech (such as intonation fluctuations and emotional intensity) deviate significantly from the source speech. Summary of the Invention

[0007] The purpose of this invention is to provide an efficient training and high-performance speech conversion model based on an acoustic model and a decoupled architecture of vocoder, thereby solving the aforementioned problems in the prior art.

[0008] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0009] An efficient training and high-expressive speech conversion model based on a decoupled architecture of acoustic model and vocoder, including an acoustic model and a vocoder;

[0010] The acoustic model includes,

[0011] Speaker encoder: used to obtain the speaker vector s based on the Mel spectrogram of the input data; used to obtain the speaker vector s_hat based on the Mel spectrogram obtained by the Mel decoder;

[0012] Content encoder: Used to calculate and obtain the prior distribution z_p based on the content features of the input data and the speaker vector s;

[0013] Normalized flow: used to calculate KL loss by matching prior distribution z_p and posterior distribution z;

[0014] Posterior encoder: used to encode the posterior distribution z based on the linear spectrogram of the input data and the speaker vector s;

[0015] Mel decoder: used to decode and obtain the Mel spectrogram based on the posterior distribution z and the speaker vector s;

[0016] Discriminator: Used to calculate adversarial loss based on the Mel spectrogram obtained from the Mel decoder;

[0017] The vocoder is used to obtain speech audio based on the Mel spectrogram obtained from the Mel decoder.

[0018] Preferably, the training process for the speech conversion model is as follows:

[0019] The Mel spectrogram of the training data is processed by a speaker encoder to obtain a speaker vector s. The linear spectrogram of the training data is input into a posterior encoder and encoded with the speaker vector s to obtain a posterior distribution z. The posterior distribution z is processed by a Mel decoder and decoded with the speaker vector s to obtain a Mel spectrogram. The content features of the training data are processed by a content encoder and encoded with the speaker vector s to obtain a prior distribution z_p.

[0020] The Mel spectrogram obtained from the Mel decoder is input into the discriminator to calculate the adversarial loss. The prior distribution z_p and the posterior distribution z are matched through a normalized flow, and the KL loss is calculated. The Mel loss is calculated using the Mel spectrogram obtained from the Mel decoder and the Mel spectrogram of the training data. The Mel spectrogram obtained from the Mel decoder is input into the speaker encoder to obtain the speaker vector s_hat of the generated spectrogram. The speaker loss is calculated using the speaker vector s and the speaker vector s_hat.

[0021] During training, adversarial loss, KL loss, Mel loss, and speaker loss are used together to optimize the speech conversion model parameters.

[0022] Preferably, the reasoning process of the speech conversion model is as follows:

[0023] The input audio content features and the target timbre's Mel spectrogram are processed by a speaker encoder to obtain a speaker vector. The content features are input into the content encoder and the speaker vector is added. Then, the content features are processed by a normalized stream to obtain latent variables. The latent variables are processed by a Mel decoder and the speaker vector is added to obtain the Mel spectrogram. Finally, the Mel spectrogram is input into a vocoder to obtain the speech audio.

[0024] Preferably, the Mel decoder includes a one-dimensional convolutional layer, a gating unit, and a one-dimensional convolutional layer; the operation of the Mel decoder is as follows:

[0025] The latent variables are passed through a one-dimensional convolutional layer to obtain latent features, which are then input together with the speaker vector into a gating unit for feature fusion. The fused features are then passed through another one-dimensional convolutional layer to obtain the output. This process is repeated N times to output the Mel spectrogram.

[0026] Preferably, the discriminator includes a one-dimensional convolution with multiple receptive fields and a two-dimensional convolution with multiple receptive fields. The one-dimensional convolution with multiple receptive fields can effectively model short-term and long-term temporal dependencies, while the two-dimensional convolution with multiple receptive fields can extract local structural features that combine time and frequency. The discriminator operates as follows:

[0027] The real Mel spectrogram and the synthesized Mel spectrogram are respectively input into the discriminator. The feature maps are extracted by one-dimensional convolution with multiple different receptive fields and two-dimensional convolution with multiple different receptive fields. The adversarial loss is calculated by comparing the feature maps of different receptive fields.

[0028] Preferably, the content encoder of the speech conversion model introduces fundamental frequency and energy as auxiliary inputs to enhance speech expressiveness; the training process of the speech conversion model after introducing fundamental frequency and energy is as follows:

[0029] The fundamental frequency information of the training data is extracted using the FCPE algorithm. The Mel spectrogram of each frame is summed along the frequency dimension to extract its frame-level energy. The Mel spectrogram of the training data is processed by a speaker encoder to obtain a speaker vector s. The linear spectrogram of the training data is input into a posterior encoder and encoded with the speaker vector s to obtain a posterior distribution z. The posterior distribution z is then processed by a Mel decoder and decoded with the speaker vector s, fundamental frequency, and energy information to obtain the Mel spectrogram. The content features of the training data are processed by a content encoder and encoded with the speaker vector s, fundamental frequency, and energy information to obtain a prior distribution z_p.

[0030] The Mel spectrogram obtained from the Mel decoder is input into the discriminator to calculate the adversarial loss. z_p and z are matched through a normalized flow and the KL loss is calculated. The Mel spectrogram obtained from the Mel decoder is compared with the real Mel spectrogram of the training data to calculate the Mel loss. Energy information is extracted from the Mel spectrogram obtained from the Mel decoder and compared with the real energy to calculate the energy loss. The Mel spectrogram obtained from the Mel decoder is input into the speaker encoder to obtain the speaker vector s_hat of the generated spectrogram. The speaker loss is calculated by comparing s and s_hat.

[0031] During training, adversarial loss, KL loss, Mel loss, energy loss, and speaker loss are used to optimize the speech conversion model parameters.

[0032] Preferably, the reasoning process of the speech conversion model after introducing fundamental frequency and energy is as follows:

[0033] The input audio content features and the target timbre's Mel spectrogram are processed by a speaker encoder to obtain a speaker vector. The content features are input into the content encoder, and the speaker vector, fundamental frequency, and energy information are added. Then, they are processed by a normalized stream to obtain latent variables. The latent variables are processed by a Mel decoder, and the speaker vector, fundamental frequency, and energy information are added to obtain the Mel spectrogram. Finally, the Mel spectrogram is input into a vocoder to obtain the speech audio.

[0034] Preferably, the formula for calculating the energy loss is as follows:

[0035]

[0036] Among them, L energy E represents the energy loss value; T0 is the frame number of the Mel spectrum; E pred [t0] represents the energy of the generated Mel spectrum; E gt [t0] represents the true Mel spectrum energy of the training data.

[0037] Preferably, the fundamental frequency can be adaptively adjusted according to the speaking characteristics of the target speaker to ensure that the converted speech conforms to the natural speech style of the target speaker; the adaptive adjustment process of the fundamental frequency is as follows:

[0038] S1. Audio frame extraction:

[0039]

[0040] in, The fundamental frequency of the spoken portion of the target speaker's audio. The audible portion of the target speaker's audio; The fundamental frequency of the audible portion of the input source speech audio; t represents the audible portion of the input source speech audio; t represents the time-domain frame of the spectrogram.

[0041] S2. Extract the median of the logarithmic field:

[0042]

[0043] Where, μ ref μ is the logarithmic midpoint of the target speaker's audio. src is the logarithmic median of the input source speech audio; median is the median operation; log is the logarithm operation; ∈ is a small constant;

[0044] S3, Automatic Baseband Adjustment:

[0045]

[0046] in, This is the automatically adjusted logarithmic domain fundamental frequency value; for the fundamental frequency For the portion greater than 1, perform an overall linear adjustment on the fundamental frequency. The portion less than 1 is retained; The fundamental frequency value in the logarithmic field;

[0047] S4, Exponential Restoration:

[0048]

[0049] in, is the fundamental frequency after logarithmic field restoration; exp is the exponentiation operation.

[0050] Preferably, step S4 is followed by...

[0051] S5, semitone transposition:

[0052]

[0053] in, s is the transposed fundamental frequency value; s0 is the adjusted semitone offset value.

[0054] The beneficial effects of this invention are: 1. Improved training efficiency and debugging flexibility: This invention adopts a decoupled training method for the acoustic model and vocoder, allowing the acoustic model to focus on generating Mel spectrograms, while the vocoder independently handles speech reconstruction, avoiding redundant computation and stability issues caused by joint training. This design significantly improves training efficiency and facilitates module-level debugging, effectively improving the efficiency of positioning and system maintainability during development. 2. Enhanced system scalability and adaptability: The decoupled architecture supports flexible replacement and independent optimization of the acoustic model and vocoder, adapting to diverse deployment needs (such as lightweight or high-fidelity scenarios). 3. Significantly improved speech naturalness and sound quality: The acoustic model introduces a latent variable decoding mechanism, making the mapping between the latent space and acoustic features more accurate, improving the consistency of speech content and the transfer effect of target timbre. Combined with a discriminator oriented towards the Mel spectrogram, the adversarial mechanism improves the quality of the Mel spectrogram, effectively improving the problem of high-frequency detail blurring in traditional methods, and enhancing the clarity and naturalness of the final speech. 4. Ensure high-quality audio output and system robustness: Utilize pre-trained high-performance vocoders (such as BigVGAN and VOCOS) for Mel spectrogram-to-audio reconstruction, improving speech synthesis quality while avoiding instability factors in end-to-end vocoder training, ensuring system stability and consistency under different data and scenarios. 5. Enhance speech emotion expression and prosodic modeling capabilities: Introduce fundamental frequency and energy information as auxiliary inputs into the acoustic model, enabling the model to effectively capture prosodic features such as pitch changes and intonation strength, improving the ability to restore semantic information such as intonation and emotion, and significantly enhancing the emotional expressiveness of speech. 6. Provide automatic / manual prosodic control, enhancing interactivity and personalization: The designed fundamental frequency control module supports automatic adaptation to the target speaker's style, while allowing users to manually edit the pitch curve and energy profile, satisfying personalized control over intonation, rhythm, emotion, etc., enhancing system controllability and user interaction experience. Attached Figure Description

[0055] Figure 1 This is a flowchart illustrating the sound conversion process;

[0056] Figure 2 This is a flowchart of the workflow of existing voice conversion technology;

[0057] Figure 3 This is a flowchart of the speech conversion model in this embodiment of the invention;

[0058] Figure 4 This is a flowchart of the Mel decoder in an embodiment of the present invention;

[0059] Figure 5 This is a flowchart of the discriminator in an embodiment of the present invention;

[0060] Figure 6This is a flowchart illustrating the workflow of the speech conversion model in this embodiment of the invention after incorporating fundamental frequency and energy information;

[0061] Figure 7 This is a flowchart of the adaptive adjustment of the base frequency in an embodiment of the present invention. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0063] Example 1

[0064] This embodiment provides an efficient and high-performance speech conversion model based on a decoupled architecture of acoustic model and vocoder. This speech model decouples the acoustic model (i.e., the acoustic modeling module) from the vocoder. The acoustic model is responsible for generating a Mel spectrogram based on the input audio content and the target speaker, while the vocoder, as an independent module, reconstructs the audio waveform from the Mel spectrogram. This decoupling design significantly improves the training efficiency of the speech conversion model, as well as the flexibility and module controllability during training, making the speech conversion model more operable in debugging, training, and deployment. Simultaneously, this structure facilitates the introduction of multiple supervision mechanisms during the acoustic modeling stage and allows for independent optimization or replacement of the vocoder module while keeping the acoustic model parameters unchanged, thereby enhancing the system's scalability and adaptability.

[0065] The speech model specifically includes the acoustic model and the vocoder;

[0066] 1. The acoustic model includes,

[0067] (1) Speaker encoder: used to obtain speaker vector s based on the Mel spectrogram of the input data; used to obtain speaker vector s_hat based on the Mel spectrogram obtained by the Mel decoder;

[0068] (2) Content encoder: used to calculate and obtain the prior distribution z_p based on the content features of the input data and the speaker vector s;

[0069] (3) Normalized flow: used to calculate KL loss by matching prior distribution z_p and posterior distribution z;

[0070] (4) Posterior encoder: used to encode the posterior distribution z based on the linear spectrum of the input data and the speaker vector s;

[0071] (5) Mel decoder: used to decode and obtain the Mel spectrogram based on the posterior distribution z and the speaker vector s;

[0072] (6) Discriminator: used to calculate the adversarial loss based on the Mel spectrogram obtained from the Mel decoder;

[0073] 2. Vocoder: Used to obtain speech audio based on the Mel spectrogram obtained from the Mel decoder.

[0074] In this embodiment, as Figure 3 As shown, during the training of the speech conversion model: the Mel spectrogram of the training data is processed by the speaker encoder to obtain the speaker vector s; the linear spectrogram of the training data is input into the posterior encoder and encoded with the speaker vector s to obtain the posterior distribution z; the posterior distribution z is processed by the Mel decoder and decoded with the speaker vector s to obtain the Mel spectrogram; the content features of the training data are processed by the content encoder and encoded with the speaker vector s to obtain the prior distribution z_p. During the optimization of the speech conversion model, the Mel spectrogram obtained from the Mel decoder is input into the discriminator to calculate the adversarial loss; z_p and z are matched using a normalized flow, and the KL loss is calculated; the Mel spectrogram obtained from the Mel decoder is compared with the true Mel spectrogram of the training data to calculate the Mel loss; the Mel spectrogram obtained from the Mel decoder is input into the speaker encoder to obtain the speaker vector s_hat of the generated spectrogram; s and s_hat are used to calculate the speaker loss; the speech conversion model is optimized by the adversarial loss, KL loss, Mel loss, and speaker loss.

[0075] During inference, the speech conversion model inputs the content features of the audio and the Mel spectrogram of the target timbre. The Mel spectrogram of the target timbre is processed by a speaker encoder to obtain a speaker vector s. The content features are input into the content encoder and the speaker vector s is added. Then, the content features are processed by a normalized stream to obtain a latent variable z. z is processed by a Mel decoder and the speaker vector s is added to obtain the Mel spectrogram. Finally, the Mel spectrogram is input into a vocoder to obtain the speech audio.

[0076] This invention innovatively proposes an architecture that decouples the acoustic model from the vocoder in a speech conversion system. The acoustic model focuses on generating intermediate acoustic representations (such as Mel spectrograms) based on content encoding and speaker embedding, while the vocoder, as an independent module, reconstructs the audio waveform. This structure overcomes the problems of module coupling and training limitations caused by end-to-end modeling in existing technologies, providing a structural foundation for the replacement and optimization of various vocoder schemes.

[0077] In this embodiment, a latent variable-based Mel decoder is designed in the acoustic model. This Mel decoder receives the latent variable z output by the posterior encoder and, given the speaker vector s, decodes it into a Mel spectrogram corresponding to the target speaker's timbre. The Mel decoder is as follows: Figure 4As shown, the latent variable z is passed through a one-dimensional convolutional layer to obtain latent features, which are then input together with the speaker vector s into a gating unit for feature fusion. The fused features are then passed through another one-dimensional convolutional layer to obtain the output. This process is repeated N times to output the Mel spectrogram. This design achieves a high-quality mapping between the latent space and the Mel spectrogram, which helps to improve the acoustic model's performance in content preservation and timbre conversion.

[0078] This invention designs an acoustic decoding module based on latent variable modeling, namely a Mel spectrogram decoder. This module uses latent variables extracted by the posterior encoder as input and generates corresponding acoustic features under the condition of target speaker embedding, constructing a high-quality mapping mechanism between the latent space and the acoustic space. This provides technical support for improving the accuracy of timbre transfer and the consistency of generated speech.

[0079] To address the issue that existing speech conversion technologies typically use discriminators that operate on audio waveforms and cannot directly evaluate Mel spectrograms, this embodiment innovatively proposes a discriminator structure specifically for Mel spectrogram discrimination. The discriminator structure is as follows: Figure 5 As shown, the discriminator consists of multi-receptive-field one-dimensional convolutions and multi-receptive-field two-dimensional convolutions. The multi-receptive-field one-dimensional convolutions can effectively model short-term and long-term temporal dependencies, while the multi-receptive-field two-dimensional convolutions can extract joint time-frequency local structural features. The discriminator's workflow is as follows: real and synthesized Mel spectrograms are input into the discriminator, and feature maps are extracted using multiple one-dimensional and two-dimensional convolutions with different receptive fields. The feature maps from different receptive fields are compared to calculate the adversarial loss, thereby optimizing the acoustic model. This discriminator can identify and provide feedback on the quality of the Mel spectrograms generated by the acoustic model, thereby improving the realism and expressiveness of the Mel spectrograms through adversarial training. This mechanism effectively improves the sound quality of the converted speech, especially significantly improving the reconstruction quality of the Mel spectrogram in the decoupled acoustic model and vocoder structure.

[0080] This invention addresses the problem in existing speech conversion systems where discriminators primarily operate on audio waveforms and lack supervision for discriminating intermediate acoustic features (such as Mel spectrograms). It proposes a network structure that directly discriminates based on Mel spectrograms. By introducing this discriminator and combining it with an adversarial training mechanism, the realism and expressiveness of the Mel spectrograms output by the acoustic model can be directly improved, providing higher-quality input for subsequent vocoder reconstruction.

[0081] To further improve the quality and training efficiency of speech synthesis, this embodiment introduces a pre-trained neural vocoder to reconstruct the speech waveform from the Mel spectrogram output by the acoustic model. Simultaneously, training data can be used to transfer training the pre-trained vocoder, specifically optimizing it. By employing a high-performance, pre-trained vocoder (e.g., BigvGAN, VOCOS), the naturalness and clarity of the synthesized speech can be effectively improved, while avoiding the training burden caused by the synchronous updating of vocoder parameters and the acoustic model, thus achieving a more efficient and controllable speech conversion process.

[0082] Example 2

[0083] To address the shortcomings of existing voice conversion models in handling speech conversion tasks with rich emotional content, this embodiment further improves the voice model of Embodiment 1. The improved voice model is as follows: Figure 6 As shown.

[0084] To enhance the expressiveness and naturalness of the converted speech, this embodiment introduces fundamental frequency and energy information as auxiliary inputs into the acoustic model to improve speech expression. The structure of the speech conversion model after incorporating fundamental frequency and energy information is as follows: Figure 6 As shown. During training: The FCPE algorithm is used to extract the fundamental frequency information of the training data. The Mel spectrogram of each frame is summed in the frequency dimension to extract its frame-level energy. The Mel spectrogram of the training data is passed through the speaker encoder to obtain the speaker vector s. The linear spectrogram of the training data is input into the posterior encoder and encoded with the speaker vector s to obtain the posterior distribution z. The posterior distribution z is passed through the Mel decoder and decoded with the speaker vector s, fundamental frequency, and energy information to obtain the Mel spectrogram. The content features of the training data are passed through the content encoder and encoded with the speaker vector s, fundamental frequency, and energy information to obtain the prior distribution z_p. In optimizing the speech conversion model, the Mel spectrogram obtained from the Mel decoder is input into the discriminator to calculate the adversarial loss. z_p and z are matched using a normalized flow, and the KL loss is calculated. The Mel spectrogram obtained from the Mel decoder is compared with the real Mel spectrograms from the training data to calculate the Mel loss. Energy information is extracted from the Mel spectrogram obtained from the Mel decoder and compared with the real energy to calculate the energy loss. The Mel spectrogram obtained from the Mel decoder is input into the speaker encoder to obtain the speaker vector s_hat of the generated spectrogram. The speaker loss is calculated using s and s_hat. The speech conversion model is optimized by combining the adversarial loss, KL loss, Mel loss, energy loss, and speaker loss. The energy loss uses L1 loss, calculated using the following formula.

[0085]

[0086] Among them, L energy E represents the energy loss value; T0 is the frame number of the Mel spectrum; E pred[t0] represents the energy of the generated Mel spectrum; E gt [t0] represents the true Mel spectrum energy of the training data.

[0087] During inference, the speech conversion model works as follows: the input audio content features and the Mel spectrogram of the target timbre are input. The Mel spectrogram of the target timbre is passed through the speaker encoder to obtain the speaker vector s. The content features are input into the content encoder and the speaker vector s, fundamental frequency, and energy information are added. Then, the normalized stream is passed to obtain the latent variable z. z is passed through the Mel decoder and the speaker vector s, fundamental frequency, and energy information are added to obtain the Mel spectrogram. Finally, the Mel spectrogram is input into the vocoder to obtain the speech audio.

[0088] In this embodiment, fundamental frequency and energy information directly reflect the prosodic features and emotional expression in speech. During training, they are used as conditional inputs along with content encoding and speaker vectors to guide the generation of latent variables that more closely resemble the pitch and energy variations of real speech. By introducing fundamental frequency and energy information, the speech conversion model's ability to model the emotion, tone, and intensity variations of the source speech is significantly enhanced, especially in speech conversion tasks with rich emotional content.

[0089] In the inference phase of the speech conversion model after incorporating the fundamental frequency, if the input fundamental frequency does not match the target speaker's fundamental frequency, problems such as unnatural and discordant timbre, ambiguous speaker identity, and inconsistent content expression may occur. To improve the adaptability of the speech conversion model in multi-speaker and personalized speech conversion scenarios, this invention further designs an automatic fundamental frequency adjustment module, such as... Figure 7 As shown. This module can adaptively adjust the input fundamental frequency information according to the speaking characteristics of the target speaker, ensuring that the converted speech matches the natural speech style of the target speaker. Simultaneously, to meet the needs of fine-grained control, it supports manual intervention by the user to adjust the fundamental frequency, enabling customized adjustments to intonation, emotion, etc., giving the system stronger controllability and interactivity. The adaptive fundamental frequency adjustment steps are as follows:

[0090] (1) Extraction of audio frames:

[0091]

[0092] in, The fundamental frequency of the spoken portion of the target speaker's audio. The audible portion of the target speaker's audio; The fundamental frequency of the audible portion of the input source speech audio; t represents the audible portion of the input source speech audio; t represents the time-domain frame of the spectrogram.

[0093] (2) Extracting the midpoint of the logarithmic field:

[0094]

[0095] Where, μ ref μ is the logarithmic midpoint of the target speaker's audio. src is the logarithmic median of the input source speech audio; median is the median operation; log is the logarithm operation; ∈ is a small constant;

[0096] (3) Automatic baseband adjustment:

[0097]

[0098] in, This is the automatically adjusted logarithmic domain fundamental frequency value; for the fundamental frequency For the portion greater than 1, perform an overall linear adjustment on the fundamental frequency. The portion less than 1 is retained; The fundamental frequency value in the logarithmic field;

[0099] (4) Exponential Restoration:

[0100]

[0101] in, is the fundamental frequency after logarithmic field restoration; exp is the exponentiation operation.

[0102] (5) Semitone transposition (optional): If a transposition parameter is specified, a semitone shift is applied to the audio frame.

[0103]

[0104] in, s is the transposed fundamental frequency value; s0 is the adjusted semitone offset value.

[0105] (6) Final output: Output the adjusted fundamental frequency sequence.

[0106]

[0107] Where f0 is the adjusted base frequency.

[0108] To enhance the performance of speech conversion models in handling emotionally rich speech tasks, this invention introduces fundamental frequency and energy as auxiliary input features during the acoustic modeling stage. These prosodic information, along with content encoding and speaker embedding, constitute multimodal modeling conditions, guiding latent variables to generate Mel spectrograms with more realistic speech styles, thereby improving the system's ability to model extra-semantic features such as intonation and emotion.

[0109] This invention further proposes an automatic and manual control mechanism that combines fundamental frequency and energy. This module can automatically adjust input parameters according to the characteristics of the target speaker (such as average fundamental frequency) to achieve prosody matching in cross-speaker scenarios; at the same time, it supports users to manually adjust the F0 curve and energy profile through a visual interface, giving the system fine control over speech emotion and intonation.

[0110] The acoustic model and vocoder decoupling structure, latent variable-based decoder design, and prosodic parameter (such as fundamental frequency F0 and energy) modeling and control mechanism in this invention can be directly applied to speech synthesis systems to improve the naturalness, emotional expression, and controllability of generated speech. It supports end-to-end interpretable modeling and multi-speaker / multi-style synthesis tasks. This invention can be applied to scenarios such as speech style transfer and anthropomorphic digital human speech generation. Through the control of latent variable space and prosodic parameters, it achieves style personalization, tone adjustment, and multi-emotion synthesis, making it widely applicable to fields such as intelligent customer service, virtual anchors, and social voice. The intermediate representations output by the acoustic model in this invention (such as Mel spectrograms or latent variables) can also serve as control signals for visual synthesis modules (such as facial animation), enabling speech-driven lip-movement synthesis (Talking Head Generation) or simultaneous speech-video generation. This structure, through acoustic-visual decoupling, enhances the system's modularity, replaceability, and training optimization flexibility. The decoupling structure and pre-trained vocoder of this invention can be used in low bit-rate speech compression systems, transmitting Mel spectrograms or latent variables as compressed representations, which are then restored to high-fidelity speech at the receiving end. The introduction of a prosodic parameter control module can also be used for adaptive optimization of speech clarity and style adaptation in weak network environments.

[0111] The latent variable modeling mechanism and controllable prosody adjustment method proposed in this invention have a general structural design and training strategy, and support integration and extended application in related tasks with minimal modifications, such as speech style generators, emotion recognition-response systems, and speech-driven music-assisted generation systems.

[0112] By adopting the above-disclosed technical solution of this invention, the following beneficial effects are obtained:

[0113] This invention provides a highly efficient and expressive speech conversion model based on a decoupled architecture of acoustic model and vocoder, improving training efficiency and debugging flexibility. The invention employs a decoupled training method for the acoustic model and vocoder, allowing the acoustic model to focus on generating Mel spectrograms while the vocoder independently handles speech reconstruction, avoiding redundant computation and stability issues caused by joint training. This design significantly improves training efficiency and facilitates module-level debugging, effectively enhancing the efficiency of localization and system maintainability during development. It also enhances the system's scalability and adaptability: the decoupled architecture supports flexible replacement and independent optimization of the acoustic model and vocoder, adapting to diverse deployment needs (such as lightweight or high-fidelity scenarios). Furthermore, it significantly improves speech naturalness and sound quality: the acoustic model introduces a latent variable decoding mechanism, making the mapping between the latent space and acoustic features more accurate, improving speech content consistency and target timbre transfer. Combined with a discriminator oriented towards the Mel spectrogram, an adversarial mechanism improves the quality of the Mel spectrogram, effectively addressing the problem of high-frequency detail blurring in traditional methods, and enhancing the clarity and naturalness of the final speech. Ensuring high-quality audio output and system robustness: Utilizing pre-trained high-performance vocoders (such as BigVGAN and VOCOS) for Mel spectrogram-to-audio reconstruction improves speech synthesis quality while avoiding instability factors in end-to-end vocoder training, ensuring system stability and consistency across different data and scenarios. Enhancing speech emotion expression and prosodic modeling capabilities: Introducing fundamental frequency and energy information as auxiliary inputs into the acoustic model enables it to effectively capture prosodic features such as pitch changes and intonation strength, improving the ability to reconstruct semantic-non-semantic information such as intonation and emotion, and significantly enhancing the emotional expressiveness of speech. Providing automatic / manual prosodic control to enhance interactivity and personalization: The designed fundamental frequency control module supports automatic adaptation to the target speaker's style, while allowing users to manually edit the pitch curve and energy profile, satisfying personalized control over intonation, rhythm, and emotion, enhancing system controllability and user interaction experience.

[0114] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A highly efficient and expressive speech conversion model based on a decoupled architecture of acoustic model and vocoder, characterized in that: Includes acoustic models and vocoders; The acoustic model includes, Speaker encoder: used to obtain the speaker vector s based on the Mel spectrogram of the input data; used to obtain the speaker vector s_hat based on the Mel spectrogram obtained by the Mel decoder; Content encoder: Used to calculate and obtain the prior distribution z_p based on the content features of the input data and the speaker vector s; Normalized flow: used to calculate KL loss by matching prior distribution z_p and posterior distribution z; Posterior encoder: used to encode the posterior distribution z based on the linear spectrogram of the input data and the speaker vector s; Mel decoder: used to decode and obtain the Mel spectrogram based on the posterior distribution z and the speaker vector s; Discriminator: Used to calculate adversarial loss based on the Mel spectrogram obtained from the Mel decoder; The vocoder is used to obtain speech audio based on the Mel spectrogram obtained from the Mel decoder. The training process for the speech conversion model is as follows: The Mel spectrogram of the training data is processed by a speaker encoder to obtain a speaker vector s. The linear spectrogram of the training data is input into a posterior encoder and encoded with the speaker vector s to obtain a posterior distribution z. The posterior distribution z is processed by a Mel decoder and decoded with the speaker vector s to obtain a Mel spectrogram. The content features of the training data are processed by a content encoder and encoded with the speaker vector s to obtain a prior distribution z_p. The Mel spectrogram obtained from the Mel decoder is input into the discriminator to calculate the adversarial loss. The prior distribution z_p and the posterior distribution z are matched through a normalized flow, and the KL loss is calculated. The Mel loss is calculated using the Mel spectrogram obtained from the Mel decoder and the Mel spectrogram of the training data. The Mel spectrogram obtained from the Mel decoder is input into the speaker encoder to obtain the speaker vector s_hat of the generated spectrogram. The speaker loss is calculated using the speaker vector s and the speaker vector s_hat. During training, adversarial loss, KL loss, Mel loss, and speaker loss are used together to optimize the speech conversion model parameters. The reasoning process of the speech conversion model is as follows: The input audio content features and the target timbre Mel spectrogram are processed by the speaker encoder to obtain the speaker vector. The content features are input into the content encoder and the speaker vector is added. Then, the normalized stream is used to obtain the latent variables. The latent variables are processed by the Mel decoder and the speaker vector is added to obtain the Mel spectrogram. Finally, the Mel spectrogram is input into the vocoder to obtain the speech audio. The Mel decoder includes a one-dimensional convolutional layer, a gating unit, and a one-dimensional convolutional layer; the operation of the Mel decoder is as follows: The latent variables are passed through a one-dimensional convolutional layer to obtain latent features, which are then input together with the speaker vector into a gating unit for feature fusion. The fused features are then passed through a one-dimensional convolutional layer to obtain the output. The above process is repeated N times to output the Mel spectrogram. The discriminator includes a one-dimensional convolution with multiple receptive fields and a two-dimensional convolution with multiple receptive fields. The one-dimensional convolution with multiple receptive fields can effectively model the temporal dependencies between short and long time periods, while the two-dimensional convolution with multiple receptive fields can extract local structural features that combine time and frequency. The discriminator operates as follows: The real Mel spectrogram and the synthesized Mel spectrogram are respectively input into the discriminator. The feature maps are extracted by one-dimensional convolution with multiple different receptive fields and two-dimensional convolution with multiple different receptive fields. The adversarial loss is calculated by comparing the feature maps of different receptive fields.

2. The efficient training and high-expressive speech conversion model based on the decoupled architecture of acoustic model and vocoder as described in claim 1, characterized in that: The content encoder of the speech conversion model introduces fundamental frequency and energy as auxiliary inputs to enhance speech expressiveness; the training process of the speech conversion model after introducing fundamental frequency and energy is as follows: The fundamental frequency information of the training data is extracted using the FCPE algorithm. The Mel spectrogram of each frame is summed along the frequency dimension to extract its frame-level energy. The Mel spectrogram of the training data is processed by a speaker encoder to obtain a speaker vector s. The linear spectrogram of the training data is input into a posterior encoder and encoded with the speaker vector s to obtain a posterior distribution z. The posterior distribution z is then processed by a Mel decoder and decoded with the speaker vector s, fundamental frequency, and energy information to obtain the Mel spectrogram. The content features of the training data are processed by a content encoder and encoded with the speaker vector s, fundamental frequency, and energy information to obtain a prior distribution z_p. The Mel spectrogram obtained from the Mel decoder is input into the discriminator to calculate the adversarial loss. z_p and z are matched through a normalized flow and the KL loss is calculated. The Mel spectrogram obtained from the Mel decoder is compared with the real Mel spectrogram of the training data to calculate the Mel loss. Energy information is extracted from the Mel spectrogram obtained from the Mel decoder and compared with the real energy to calculate the energy loss. The Mel spectrogram obtained from the Mel decoder is input into the speaker encoder to obtain the speaker vector s_hat of the generated spectrogram. The speaker loss is calculated by comparing s and s_hat. During training, adversarial loss, KL loss, Mel loss, energy loss, and speaker loss are used to optimize the speech conversion model parameters.

3. The efficient training and high-expressive speech conversion model based on the decoupled architecture of acoustic model and vocoder as described in claim 2, characterized in that: The reasoning process of the speech conversion model after introducing fundamental frequency and energy is as follows: The input audio content features and the target timbre's Mel spectrogram are processed by a speaker encoder to obtain a speaker vector. The content features are input into the content encoder, and the speaker vector, fundamental frequency, and energy information are added. Then, they are processed by a normalized stream to obtain latent variables. The latent variables are processed by a Mel decoder, and the speaker vector, fundamental frequency, and energy information are added to obtain the Mel spectrogram. Finally, the Mel spectrogram is input into a vocoder to obtain the speech audio.

4. The efficient training and high-expressive speech conversion model based on the decoupled architecture of acoustic model and vocoder as described in claim 3, characterized in that: The formula for calculating the energy loss is as follows: Among them, L energy E represents the energy loss value; T0 is the frame number of the Mel spectrum; E pred [t0] represents the energy of the generated Mel spectrum; E gt [t0] represents the true Mel spectrum energy of the training data.

5. The efficient training and high-expressive speech conversion model based on the decoupled architecture of acoustic model and vocoder as described in claim 4, characterized in that: The fundamental frequency can be adaptively adjusted according to the speaking characteristics of the target speaker to ensure that the converted speech conforms to the natural speech style of the target speaker; the adaptive adjustment process of the fundamental frequency is as follows: S1. Audio frame extraction: in, The fundamental frequency of the spoken portion of the target speaker's audio. The audible portion of the target speaker's audio; The fundamental frequency of the audible portion of the input source speech audio; t represents the audible portion of the input source speech audio; t represents the time-domain frame of the spectrogram. S2. Extract the median of the logarithmic field: Where, μ ref μ is the logarithmic midpoint of the target speaker's audio. src is the logarithmic median of the input source speech audio; median is the median operation; log is the logarithm operation; ∈ is a small constant; S3, Automatic Baseband Adjustment: in, This is the automatically adjusted logarithmic domain fundamental frequency value; for the fundamental frequency For the portion greater than 1, perform an overall linear adjustment on the fundamental frequency. The portion less than 1 is retained; The fundamental frequency value in the logarithmic field; S4, Exponential Restoration: in, is the fundamental frequency after logarithmic field restoration; exp is the exponentiation operation.

6. The efficient training and high-expressive speech conversion model based on the decoupled architecture of acoustic model and vocoder as described in claim 5, characterized in that: Step S4 is followed by S5, semitone transposition: in, s is the transposed fundamental frequency value; s0 is the adjusted semitone offset value.

Images