Zero-shot speech cloning method based on dynamic neural network and feature modulation

Abstract

This invention discloses a zero-shot speech cloning method based on a dynamic neural network and feature modulation. The method comprises extracting a speaker style vector from reference audio using a speaker style encoder; performing feature modulation on the speaker style vector using the SGF algorithm; and inputting the modulated speaker style vector into a generator; and synthesizing the target speaker audio using a dynamic neural network. This method can clone the audio of any speaker in a zero-shot scenario, synthesizing smooth, natural, and highly similar target audio.

Description

Technical Field

[0001] The present invention belongs to the technical field of speech cloning, and in particular relates to a zero-sample speech cloning method based on dynamic neural network and feature modulation. Background Art

[0002] In recent years, significant progress has been made in the field of text-to-speech (TTS), allowing people to use technology to convert text into natural and fluent audio, giving rise to a variety of practical application scenarios. In the field of education, TTS synthesis technology can be used to synthesize high-quality speech from a single speaker and create personalized learning experiences, such as customized lecture assistance tools. In terms of auxiliary tools, multilingual synthesis provides multilingual workers with a more convenient way to communicate and understand each other, enabling them to easily access a variety of information. In the entertainment and media industry, multi-speaker speech synthesis is widely used, for example, in the production of podcasts and radio programs, and even to provide vivid voice performances for characters in games. However, this also brings a new challenge: if you are given audio and text from a speaker you have never seen, how can you simulate the speaker's voice style to make it sound like the speaker's voice? This challenge is called zero-sample voice cloning.

[0003] Despite significant progress in TTS synthesis, achieving high-quality voice cloning still faces many technical challenges, such as ensuring model generalization, effectively capturing speaker characteristics, and synthesizing natural, fluent, and highly similar audio. Currently, voice cloning methods are primarily categorized into speaker adaptation and speaker encoding. Speaker adaptation involves fine-tuning and optimizing a pre-trained model based on audio samples of a target speaker, adapting it to the characteristics of the target speaker. Speaker encoding involves extracting the speaker style from the audio sample and then using the style vector as a generator to adjust the feature vectors, ensuring that the generated audio conforms to the target speaker's speech characteristics. Speaker style representation can be achieved by using a pre-trained model and jointly optimizing the speaker representation. Some pre-trained models used for speaker verification or voice conversion tasks can learn speaker style information from audio. This approach offers high performance with limited resources and strong generalization capabilities, requiring only a few seconds of target speaker audio. Although widely used in voice cloning, speaker encoding still faces challenges in achieving speaker similarity and model generalization.

[0004] At the same time, with the advancement of voice cloning technology, the risk of bad actors exploiting zero-sample voice cloning to forge voices has gradually emerged. Fake voices can be used to spread disinformation, impersonate others, and even manipulate public opinion, causing social confusion. Especially in sensitive areas such as news reporting, social media platforms, and political commentary, the dissemination of original voices can undermine the authenticity and credibility of information, and thus affect social stability. Therefore, ensuring the security of voices generated through voice cloning technology in information dissemination and preventing them from being used for malicious manipulation has become a critical issue that needs to be addressed. Summary of the Invention

[0005] To address the aforementioned issues, the present invention provides a zero-shot speech cloning method based on dynamic neural networks and feature modulation, achieving zero-shot speech cloning. The method utilizes two aspects: feature modulation and dynamic neural networks. To address the low speaker similarity and poor generalization capabilities of speaker encoding methods, the feature modulation component extracts speaker style vectors from mel-spectrograms and mel-frequency cepstral coefficients (MFCCs). Style Gating-Film is then used to modulate the feature sequence to synthesize the target speaker's audio. To improve the quality of short sentence synthesis, experiments were conducted in a variance adapter. By evaluating the loss calculation for different phoneme sequence lengths, two predictors with different structures were used to predict the pitch, energy, and duration of phoneme sequences of varying lengths.

[0006] To achieve the above object, the technical solution adopted by the present invention is as follows:

[0007] A zero-sample speech cloning method based on dynamic neural network and feature modulation, the method comprising the following steps:

[0008] The speaker style encoder extracts the speaker style vector from the reference audio;

[0009] Perform feature modulation on the speaker style vector based on the SGF algorithm, and input the modulated speaker style vector into the generator;

[0010] The generator synthesizes target speaker audio based on a dynamic neural network.

[0011] Furthermore, the speaker style encoder includes a Mel style encoder and an MFCC style encoder, which are used to extract speaker feature style vectors contained in the Mel spectrogram and the Mel frequency cepstral coefficient respectively, and concatenate the extracted speaker feature style vectors to obtain a final speaker style vector.

[0012] Furthermore, in the speaker style encoder;

[0013] The MFCC style encoder uses a bidirectional LSTM to process the MFCC data to extract the temporal changes of the MFCC features and the dependencies in the sequence; performs self-attention calculations on the extracted MFCC features; and averages the output of the self-attention mechanism along the time axis to obtain a one-dimensional feature style vector with the same dimension as the Mel style encoder output.

[0014] The one-dimensional style vector output by the Mel style encoder is concatenated with the one-dimensional feature style vector output by the MFCC style encoder to obtain the final style vector.

[0015] Furthermore, the feature modulation of the speaker style vector based on the SGF algorithm includes:

[0016] set up represents the phoneme sequence feature vector, where Represents the dimension of the phoneme sequence feature vector, for the phoneme sequence feature vector To standardize:

[0017] ,

[0018] ,

[0019] ,

[0020] Where, represents the normalized feature vector, represents the mean of the eigenvector, represents the variance of the eigenvector, represents the feature vector index, is the dimension of the feature vector;

[0021] The final style vector is modulated by four parameters:

[0022] ,

[0023] ,

[0024] represents the first scaling factor obtained by processing the final style vector through the hyperbolic tangent function; represents the bias factor obtained by processing the final style vector through the hyperbolic tangent function; Represents the baseline feature obtained by processing the final style vector through the hyperbolic tangent function, which is used to balance and ; Represents the dynamic adjustment factor obtained by processing the final style vector through the Sigmoid function;

[0025] Four parameters are applied to the normalized feature vector via a weighting operation:

[0026] ,

[0027] ,

[0028] ,

[0029] Where, Represents the feature sequence after fusion of style vectors.

[0030] Furthermore, the generator includes a phoneme encoder, a variance adapter and a mel-spectrogram decoder, wherein the phoneme encoder is used to convert a phoneme embedding sequence into a phoneme hidden sequence containing the speaker's personalized speech style characteristics; the variance adapter is used to add pitch and energy information to the phoneme hidden sequence, and adjust the length of the phoneme hidden sequence according to the predicted duration of each phoneme; the mel-spectrogram decoder is used to inject the style vector into the adjusted phoneme hidden sequence and convert it into a mel-spectrogram sequence.

[0031] Furthermore, the variance adapter includes different predictors corresponding to long phoneme sequences and short phoneme sequences.

[0032] Furthermore, the predictor architecture for long phoneme sequences includes two layers of one-dimensional convolutional networks with ReLU activation, each network followed by a normalization and dropout layer, and a linear layer connected after the last normalization and dropout layer; the predictor architecture for short phoneme sequences includes two layers of one-dimensional convolutional networks with ReLU activation, each network followed by a normalization and dropout layer, and a one-dimensional convolutional network connected after the last normalization and dropout layer.

[0033] Furthermore, by minimizing the Mel-spectrogram reconstruction loss , duration prediction loss , Energy prediction loss , pitch prediction loss Optimize the network model consisting of the speaker encoder and generator.

[0034] The beneficial effects of the present invention are:

[0035] This paper proposes a multi-speaker adaptive TTS model with high similarity and generalization ability: the Dynamic Style Speech Synthesis Model (DS-TTS), which can perform speech cloning in zero-shot mode; proposes an innovative dual-style encoder architecture and Style Gating-Film to filter out information that has an adverse effect on learning during training, significantly improving the ability to capture and express speaker style; and proposes a dynamic neural network method based on sample length to more flexibly adapt to input changes, thereby performing better when processing texts of different lengths. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is the overall framework diagram of the zero-sample speech cloning method based on dynamic neural network and feature modulation of the present invention;

[0037] Figure 2 This is a detailed diagram of the network components of the present invention. DETAILED DESCRIPTION

[0038] The present invention will be further described below with reference to the accompanying drawings and examples.

[0039] As mentioned above, with the rapid development of text-to-speech (TTS) models, the demand for personalized audio generation is increasing. As a subtask of TTS, the goal of the zero-shot voice cloning task is to synthesize the target speaker's audio, given a single audio file and arbitrary text, without ever seeing the speaker. Although existing voice cloning models can clone the voices of unseen speakers, they still produce some unnatural and poorly similar audio. To enhance the stylistic properties of unseen speakers and the generalization ability of the model, this paper introduces the DS-TTS model. This model uses two style encoders to extract the speaker's style vector as conditional information, which is then input into the generator via Style Gating-Film (SGF). Furthermore, to address the poor synthesis performance of short sentences, this paper designs a dynamic neural network in the variance adapter, using a linear layer structure to ensure efficient processing of long phoneme sequences and a convolutional layer structure to improve the synthesis performance of short phoneme sequences. Experiments on the VCTK dataset show that the proposed model performs well on the zero-shot voice cloning task compared to state-of-the-art models.

[0040] like Figure 1Figure 2 shows the overall framework of the zero-shot speech cloning method based on dynamic neural networks and feature modulation. StyleSpeech is used as the model's foundational framework. The model primarily consists of a speaker style encoder and a generator. The Mel-style encoder and MFCC style encoder extract speaker style vectors from audio. The generator integrates these style vectors into the text-to-speech (TTS) architecture via Style Gating-Film (SGF) to synthesize the target speaker's audio. In this process, incorporating the speaker's speaking style into the TTS architecture is crucial. Furthermore, this invention introduces a dynamic neural network as an innovative approach to improve speaker generalization in short text synthesis. The dynamic neural network can dynamically select different paths or processing modes based on input features or conditions. This approach more accurately captures the specific speech characteristics of each speaker, thereby improving the naturalness of the synthesized audio and speaker similarity.

[0041] refer to Figure 2 As shown, the various parts of the network framework of the present invention are introduced below.

[0042] 1. Feature modulation;

[0043] In this method, audio is converted into a mel-spectrogram and MFCCs as input. The speaker's style is then extracted using a mel-style encoder and an MFCC style encoder. This encoder then concatenates the speaker's style vector to produce a final speaker style vector. This final speaker style vector captures individual characteristics such as the speaker's pitch, speaking rate, intonation, and pronunciation habits. Specifically, the mel-style encoder extracts speaker features from the mel-spectrogram, while the MFCC style encoder extracts speaker features from the mel-frequency cepstral coefficients (MFCCs).

[0044] The style encoder designed by the present invention includes the following three parts:

[0045] LSTM: Use bidirectional LSTM to process MFCC data to capture the temporal changes of MFCC and the dependencies in the sequence.

[0046] Multi-head self-attention mechanism: Perform self-attention calculation on the extracted MFCC features to better understand the global context information.

[0047] Average pooling: The output of the self-attention mechanism is averaged along the time axis to obtain a one-dimensional style vector with the same dimension as the Mel-style encoder output. Finally, the outputs of the two are concatenated to obtain the final style vector.

[0048] The final style vector will be input into the feedforward transformer block (FFT block) of the phoneme encoder and mel-spectrogram decoder respectively. The FFT block is an important network architecture in the phoneme encoder and mel-spectrogram decoder.

[0049] The generator consists of three parts: a phoneme encoder, a variance adapter, and a mel-spectrogram decoder. The phoneme encoder converts a phoneme embedding sequence into a phoneme latent sequence that contains the speaker's personalized speech style characteristics. The variance adapter predicts the pitch, energy, and duration of each phoneme. It adds pitch and energy information to the phoneme latent sequence and then adjusts the length of the phoneme latent sequence based on the predicted duration of each phoneme. Finally, the mel-spectrogram decoder injects the final style vector into the adjusted phoneme latent sequence and converts it into a mel-spectrogram sequence.

[0050] Generally, the style vector can be fused with feature information through conditional generation methods such as splicing, direct addition, and Style-Adaptive Layer Norm to obtain a hidden sequence of phonemes containing the speaker's personalized speech style features. Among them, the splicing and direct addition methods may cause confusion of semantic information and fail to accurately convey subtle changes in speaking style. Although Style-Adaptive Layer Norm can adaptively scale and offset feature vectors according to the style vector, it cannot filter out information that may have a negative impact on learning. Therefore, the present invention improves the Style-Adaptive Layer Norm method and proposes Style Gating-FiLM (SGF), which accepts the final style vector w and modulates the feature information. SGF introduces intermediate parameters on the basis of FiLM and , filtering out some information that has a negative impact on learning, so that the model can more finely control the mean and variance. represents the phoneme sequence feature vector, where is the dimension of the feature vector. First, the feature vector of the phoneme sequence To standardize:

[0051] ,

[0052] ,

[0053] ,

[0054] is the eigenvector, is the normalized eigenvector, is the mean of the eigenvectors, is the variance of the eigenvector, represents the feature vector index, is the dimension of the feature vector. Represents the feature vector Middle elements, ranging from 1 to N.

[0055] The final style vector is then modulated by four parameters:

[0056] ,

[0057] ,

[0058] , represents the final style vector processed by the hyperbolic tangent function , get the first scaling factor.

[0059] Represents the final style vector processed by the hyperbolic tangent function, which has the same effect as Similarly, it can be called a bias factor, which is used to provide flexibility in feature modulation and ensure diversity in the final result.

[0060] , represents the final style vector processed by the hyperbolic tangent function, and obtains the benchmark feature, which is used to provide an alternative to ensure that there is a consistent feature representation during the adjustment process, which can balance and .

[0061] , represents the final style vector processed by the Sigmoid function, the output value is in [0,1], as a dynamic adjustment factor, control 、 and The combination ratio of can realize the adaptive adjustment of the eigenvector and affect the overall modulation effect.

[0062] Finally, the modulated parameters are applied to the normalized feature vector through a weighted operation to achieve stylized control of the input feature vector:

[0063] ,

[0064] ,

[0065] The above two formulas give and It can increase the flexibility of adjusting the style vector, so that the same style vector w can adjust the feature vector More delicate control;

[0066] ,

[0067] Represents the feature sequence after fusion of style vectors.

[0068] This mechanism aims to and Scaling and offsetting normalized features to achieve adaptive adjustment of features. and Additional modulation flexibility is provided, enabling the modulation process to balance two different parameters, thereby achieving more fine-grained semantic representation and style transfer.

[0069] 2. Dynamic neural network based on phoneme sequence length;

[0070] While current models can effectively synthesize text content from most target speakers, they perform poorly when processing short text during testing. To address this challenge, the present invention proposes a dynamic neural network based on phoneme sequence length. Different predictors are designed in the variance adapter to handle short and long phoneme sequences, thereby rationally allocating computing resources based on the length of the input phoneme sequence. This dynamic architecture not only reduces redundant computation for simple samples but also maintains its representational power when recognizing complex samples. This strategy allows for more efficient optimization of voice cloning results, especially when processing short text. This approach not only improves the quality of synthesized speech but also enhances the model's ability to capture the target speaker's voice characteristics.

[0071] In the present invention, sentences with a phoneme sequence length less than or equal to 85 are classified as short phoneme sequences, and sentences with a phoneme sequence length greater than 85 are classified as long phoneme sequences. In the experiment, the MSE loss is used to calculate the predicted values ​​of pitch, energy, and duration. and the true value The difference between the two is that the original predictor of the basic framework StyleSpeech performs well when cloning long phoneme sequences. Therefore, the present invention designs a pitch, energy and duration predictor framework specifically for short phoneme sequences:

[0072] ;

[0073] is the i-th characteristic component of the predicted value; is the i-th feature component of the true value, and N is the dimension of the feature vector;

[0074] Represents the predicted value of pitch, energy, and duration and the true value The mean square error of .

[0075] In the original predictor, the model uses a linear layer for projection. It has a strong fitting ability and can learn more complex mapping relationships, thus better adapting to complex data structures. However, in short sentence synthesis, using a linear layer for projection may lead to overfitting of the model. Therefore, the present invention adopts a convolutional layer with local connection and weight sharing features to better adapt to the characteristics of short sentences. Finally, the modified variance adapter architecture is as follows Figure 2 As shown in Figure 3, the predictor architecture for pitch, energy, and duration for long phoneme sequences consists of two layers of one-dimensional convolutional networks with ReLU activations, each followed by a normalization and dropout layer, and a linear layer connected after the last normalization and dropout layer. The predictor architecture for short phoneme sequences consists of two layers of one-dimensional convolutional networks with ReLU activations, each followed by a normalization and dropout layer, and a one-dimensional convolutional network connected after the last normalization and dropout layer.

[0076] During training, by minimizing the Mel-spectrogram reconstruction loss , duration prediction loss , Energy prediction loss , pitch prediction loss To optimize the model, the Mel spectrogram reconstruction loss The MAE loss function is used, and the MSE loss function is used for other losses.

[0077] The total losses are as follows:

[0078] .

[0079] During training, we used the clean and noise-free train-clean-100 and train-clean-360 datasets from the LibriTTS dataset, which contain data from 1,148 speakers. We also used the VCTK dataset for evaluation on unseen speakers, which contains 108 speakers.

[0080] Before training, some preprocessing is required on the audio and text data: 1) the sampling rate of the training and evaluation data is converted to 16kHz; 2) the spectrogram is extracted using an FFT size of 1024, a hop size of 256, and a window size of 1024 samples, and converted into a Mel-spectrogram with 80 frequency bins; 3) 20 MFCC coefficients are extracted from the Mel-spectrogram as features using librosa and the MFCC features are normalized with the L2 norm; 4) the audio data and text data are aligned using Montreal ForcedAligner to obtain the duration of each phoneme.

[0081] Melgan is used as the vocoder. The outputs of the Mel style encoder and MFCC style encoder are both 128-dimensional vectors, which are then concatenated into a 256-dimensional style vector. The Style Gating-FiLM layer is placed between the FFT of the phoneme encoder and the Mel spectrogram decoder. In the initialization phase, the parameters The bias is set to 1.0, and the other three parameters are 、 and The bias of is set to 0. This design is intended to ensure that the model has a certain balance in the initial stage, so as to better adapt to subsequent style modulation.

[0082] This method sets the model training batch size to 24 and uses the Adam optimizer with the following parameters: , , During training, 200,000 steps of training were performed to obtain the final optimized model for practical use.

[0083] Audio samples of 108 different speakers were randomly selected from the VCTK dataset as a test set to evaluate the model's generalization ability for unseen speakers. The model was compared with audio samples generated by several baselines, and the results of WER and SMCS are shown in Table 1. Obviously, DS-TTS performed well. Although the WER of DS-TTS was slightly higher than StyleTTS2 and XTTS v2, it was significantly better than them in terms of similarity. The SMCS performance of VALL-EX was similar to that of DS-TTS, but its WER was 0.228, which was relatively poor. This may be due to the limited amount of training data used by the authors, which only used 704 hours of English training data. The original English ASR data of LibriLight contains about 60,000 hours of unlabeled speech, which is obviously richer. Overall, the model of the present invention performs well in speaker similarity performance, while having a low word error rate and high-quality speech synthesis.

[0084] Table 1

[0085]

[0086] The baseline models involved in the table are introduced as follows:

[0087] GT: stands for True Voice.

[0088] StyleSpeech: Uses a Mel-style encoder to extract the speaker's style and incorporates it into a speech synthesis model based on FastSpeech2 through the Style-AdaptiveLayer Norm to synthesize the target speaker's audio.

[0089] Retrained StyleSpeech: To ensure the accuracy of the model’s normal training and comparison, we adjusted the batch size to 24, trained the model for 200,000 steps, and retrained the StyleSpeech model using the same dataset split.

[0090] YourTTS: A zero-shot speaker-adapted voice cloning model based on VITS, using speaker consistency loss as the loss function and evaluated using checkpoints from Coqui TTS.

[0091] VALL-EX: A cross-lingual neural encoding language model based on VALL-E that uses reference audio and text as clues to predict the acoustic symbol sequence of target audio.

[0092] StyleTTS2: A voice cloning model that uses style diffusion and adversarial training with large speech language models to achieve high-quality TTS synthesis.

[0093] Coqui AI XTTS v2: A zero-shot text-to-speech model based on Tortoise that uses the Perceiver model to prefix the GPT decoder to capture speaker characteristics.

[0094] DS-TTS: A zero-shot speech cloning model proposed in this paper generates highly similar target audio from reference audio through feature modulation and dynamic neural network.

[0095] Subjective and objective evaluation metrics are used to assess the naturalness and similarity of synthesized target speaker utterances.

[0096] 1) Subjective metrics: In terms of subjective evaluation, the mean opinion score and similarity mean opinion score are used to evaluate the quality of audio synthesis. These metrics are evaluated on a scale of 1 to 5, with 5 being the best performance.

[0097] Mean Opinion Score (MOS): Use MOS to evaluate the clarity of audio and the fluency of sentences, focusing on the presence of background noise and the naturalness and flow of prosody, including intonation, pitch, and rhythm.

[0098] Similarity Mean Opinion Score (SMOS): Use SMOS to evaluate people's subjective perception of the similarity between the synthesized audio and the reference audio.

[0099] 2) Objective indicators: In terms of objective evaluation, word error rate and speaker embedding cosine similarity are used to evaluate the quality of audio synthesis.

[0100] Word Error Rate (WER): WER is used to measure the difference between the reference text and the text recognized by the ASR model. The lower the WER value, the more accurate the pronunciation of the synthesized audio. This paper selects the basic model of the whisper library as the ASR model.

[0101] Speaker Embedding Cosine Similarity (SMCS): SMCS is used to compare the cosine similarity between the synthesized audio and the real audio. The higher the SMCS value, the more similar the speaker's voice features in the synthesized audio are. This paper uses Resemblyzer to extract speaker representations.

[0102] To further evaluate the model's performance, different baseline models were tested on short and long phoneme sequences. The results of the objective and subjective evaluations are shown in Table 2. First, the comparison of long and short phoneme sequences shows that the DS-TTS model has good adaptability when processing reference speech of any length. Second, the results of objective indicators show that DS-TTS achieves the best balance between WER and SMCS among all models. Finally, the subjective evaluation results show that, under the same small-batch data training conditions, the audio generated by DS-TTS outperforms YourTTS, StyleSpeech, and Retrained StyleSpeech in terms of MOS. However, the model's performance may be slightly inferior to StyleTTS2 and XTTS v2. This difference is understandable, as StyleTTS2 and XTTS v2 use extremely large datasets and both take much longer to train than the settings used in this paper. Most importantly, in terms of subjective similarity, DS-TTS demonstrates strong generalization ability, with the generated audio highly similar to the audio of the target speaker.

[0103] Table 2

[0104]

[0105] To evaluate performance across different genders and English accents, we selected a test set of 100 audio samples for each condition. As shown in Table 3, our model performs well in generating speech that is similar to the original speech, with no significant differences between male and female speakers, while accurately capturing the characteristics of various accents. This further validates that the features employed in this paper effectively capture the commonalities and differences between male and female speakers and various English accents, enabling the model to maintain balanced performance during the generation process.

[0106] Table 3

[0107]

[0108] Appropriate threshold selection can improve model performance, enabling it to perform better in voice cloning tasks. Based on the ablation experiment results in Table 4, the present invention evaluated multiple models with thresholds between 75 and 95. Among them, the model with a threshold of 85 performed best in the test. At a threshold of 85, the model showed a significant improvement in similarity while maintaining the lowest pronunciation recognition error rate. This may be because the threshold of 85 happens to be a kind of balance point in the dynamic neural network, which can effectively process phoneme sequences of various lengths without prematurely truncating important information or increasing the computational burden of processing overly long sequences.

[0109] Table 4

[0110]

[0111] Furthermore, a study of the effectiveness of the model components, as shown in Table 5, shows that removing MFCC features significantly degrades the quality of the generated speech. Furthermore, using only long or short predictors significantly degrades model performance. The results demonstrate that MFCC features play a crucial role in feature modulation, while the dynamic neural network further enhances the model's adaptive performance.

[0112] Table 5

[0113]

[0114] In summary, this paper proposes DS-TTS, a text-to-speech model for zero-shot speech cloning based on dynamic neural networks and feature modulation. To enhance the model's generalization, a Mel-style encoder and an MFCC-style encoder are used to extract the speaker's style vector, and the feature vector is modulated using Style Gating-Film (SGF). Furthermore, to address the poor synthesis of short phoneme sequences, the input sequence is classified according to sample size, and a dynamic neural network is used in a variance adapter to more accurately predict the speaker's pitch, energy, and duration for both short and long samples. Experimental results demonstrate that DS-TTS outperforms existing methods across multiple metrics.

[0115] The present invention, through deep neural network optimization, can achieve accurate and natural speech generation with very little audio data of the target speaker. This technology not only makes a breakthrough in the natural fluency of speech synthesis, but also correspondingly enhances the anti-tampering and anti-forgery capabilities of the generation model by optimizing the generation process and the dynamic deployment of the neural network. This anti-tampering capability can effectively induce the alarm of voice in information dissemination, ensuring that the generated voice will not be maliciously tampered with or used for malicious information dissemination in information dissemination channels such as social media and news reports. The present invention can provide a more secure speech generation technology, reduce the generation and dissemination of false voices, thereby reducing the impact of false information on public cognition, and ensure the authenticity of information and the stability of socialization.

[0116] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A zero-sample speech cloning method based on dynamic neural network and feature modulation, characterized in that: The method comprises the following steps: The speaker style encoder extracts the speaker style vector from the reference audio; The speaker style vector is modulated based on the SGF algorithm, including: set up represents the phoneme sequence feature vector, where Represents the dimension of the phoneme sequence feature vector, for the phoneme sequence feature vector To standardize: ， ， ， Where, represents the normalized feature vector, represents the mean of the eigenvector, represents the variance of the eigenvector, represents the feature vector index, is the dimension of the feature vector; The final style vector is modulated by four parameters: ， ， Where w represents the style vector, represents the first scaling factor obtained by processing the final style vector through the hyperbolic tangent function; represents the bias factor obtained by processing the final style vector through the hyperbolic tangent function; Represents the baseline feature obtained by processing the final style vector through the hyperbolic tangent function, which is used to balance and ; Represents the dynamic adjustment factor obtained by processing the final style vector through the Sigmoid function; Four parameters are applied to the normalized feature vector via a weighting operation: ， ， ， Where, Represents the feature sequence after fusion of style vectors; The modulated speaker style vector is input into a generator; the generator synthesizes the target speaker audio based on a dynamic neural network, where the dynamic neural network selects a predictor architecture based on the length of the phoneme sequence: a predictor with a linear layer is used for long phoneme sequences; a predictor with a one-dimensional convolutional layer is used for short phoneme sequences.

2. The zero-sample speech cloning method based on dynamic neural network and feature modulation according to claim 1, characterized in that: The speaker style encoder includes a Mel style encoder and an MFCC style encoder, which are used to extract the speaker characteristic style vector contained in the Mel spectrogram and the Mel frequency cepstral coefficient respectively, and concatenate the extracted speaker characteristic style vectors to obtain the final speaker style vector.

3. The zero-sample speech cloning method based on dynamic neural network and feature modulation according to claim 2, characterized in that: In the speaker style encoder; The MFCC style encoder uses a bidirectional LSTM to process the MFCC data to extract the temporal changes of the MFCC features and the dependencies in the sequence; performs self-attention calculations on the extracted MFCC features; and averages the output of the self-attention mechanism along the time axis to obtain a one-dimensional feature style vector with the same dimension as the Mel style encoder output. The one-dimensional style vector output by the Mel style encoder is concatenated with the one-dimensional feature style vector output by the MFCC style encoder to obtain the final style vector.

4. The zero-sample speech cloning method based on dynamic neural network and feature modulation according to claim 1, characterized in that: The generator includes a phoneme encoder, a variance adapter, and a mel-spectrogram decoder. The phoneme encoder is used to convert a phoneme embedding sequence into a phoneme hidden sequence containing the speaker's personalized speech style characteristics; the variance adapter is used to add pitch and energy information to the phoneme hidden sequence and adjust the length of the phoneme hidden sequence according to the predicted duration of each phoneme; the mel-spectrogram decoder is used to inject the style vector into the adjusted phoneme hidden sequence and convert it into a mel-spectrogram sequence.

5. The zero-sample speech cloning method based on dynamic neural network and feature modulation according to claim 4, characterized in that: The variance adapter includes different predictors for long phoneme sequences and for short phoneme sequences.

6. The zero-sample speech cloning method based on dynamic neural network and feature modulation according to claim 5, characterized in that: The predictor architecture for long phoneme sequences consists of two one-dimensional convolutional networks with ReLU activations, each followed by a normalization and dropout layer, and a linear layer connected after the last normalization and dropout layer; the predictor architecture for short phoneme sequences consists of two one-dimensional convolutional networks with ReLU activations, each followed by a normalization and dropout layer, and a one-dimensional convolutional network connected after the last normalization and dropout layer.

7. The zero-sample speech cloning method based on dynamic neural network and feature modulation according to claim 1, characterized in that: By minimizing the mel-spectrogram reconstruction loss , duration prediction loss , Energy prediction loss , pitch prediction loss Optimize the network model consisting of the speaker encoder and generator.

Images