Zero sample voice cloning method based on dynamic neural network and feature modulation

Abstract

The invention discloses a zero-sample voice cloning method based on a dynamic neural network and feature modulation. The method comprises the following steps: extracting a speaker style vector from a reference audio by a speaker style encoder; performing feature modulation on the speaker style vector based on an SGF algorithm, and inputting the modulated speaker style vector into a generator; the generator synthesizes the audio of the target speaker based on the dynamic neural network. According to the method, the audio of any speaker can be cloned in a zero sample situation, and the smooth and natural target audio with high similarity is synthesized.

Description

Technical Field

[0001] The invention belongs to the technical field of speech cloning, and in particular relates to a zero-sample speech cloning method based on dynamic neural network and feature modulation. Background Art

[0002] In recent years, the field of text-to-speech (TTS) has made important progress, allowing people to use technology to convert text into natural and fluent audio, which has spawned a variety of practical application scenarios. In the field of education, TTS synthesis technology can be used to synthesize high-quality speech from a single speaker and create personalized learning experiences, such as customized lecture aids. In terms of aids, multi-language synthesis provides multi-lingual workers with a more convenient way to communicate and understand, allowing them to easily obtain various information. In the entertainment and media industry, multi-speaker speech synthesis is widely used, such as producing podcasts and radio programs, and even providing vivid voice performances for characters in games. However, this also brings a new challenge: if you are given audio and text from a speaker you have never seen before, how do you simulate the speaker's voice style to make it sound like the speaker's voice? This challenge is called zero-sample voice cloning.

[0003] Although great progress has been made in the field of TTS synthesis, there are still many technical challenges in achieving high-quality voice cloning, such as the generalization ability of the model, effectively capturing speaker characteristics, and synthesizing natural, fluent and highly similar audio. At present, voice cloning methods are mainly divided into speaker adaptation methods and speaker encoding methods. Speaker adaptation methods refer to fine-tuning and optimizing the pre-trained model according to the audio samples of the target speaker to make it better adapt to the characteristics of the target speaker; speaker encoding methods refer to extracting the speaker style from the speaker audio samples, and then using the style vector as a generation condition to adjust the feature vector so that the generated audio conforms to the speech characteristics of the target speaker. Among them, speaker style representation can be achieved by using pre-trained models and jointly optimized speaker representation. Some pre-trained models for speaker verification tasks or voice conversion tasks can learn speaker style information in audio. This method has high performance with less resources, strong generalization ability, and can be applied with only a few seconds of target speaker audio. Although speaker encoding methods have been widely used in voice cloning, they still face certain challenges in speaker similarity and model generalization ability.

[0004] At the same time, with the advancement of voice cloning technology, the risk of bad actors using zero-sample voice cloning technology to forge voices has gradually emerged. False voice generation may be used to spread false information, impersonate others, and even manipulate public opinion, causing social cognitive confusion. Especially in sensitive areas such as news reporting, social platforms, and political commentary, the spread of original voices may affect the authenticity and credibility of information, and thus affect social stability. Therefore, how to ensure the security of voices generated by voice cloning technology in information dissemination and prevent them from being maliciously used for manipulation has become a key issue that needs to be solved urgently. Summary of the invention

[0005] In order to solve the above problems, the present invention provides a zero-sample speech cloning method based on dynamic neural network and feature modulation, and realizes zero-sample speech cloning. The method is deployed from two aspects: feature modulation and dynamic neural network. In order to solve the problems of low speaker similarity and poor generalization ability in speaker coding methods, in the feature modulation part, the speaker style vector is extracted from the Mel spectrogram and Mel frequency cepstral coefficient MFCC, and then Style Gating-Film is used to modulate the feature sequence to synthesize the audio of the target speaker; in order to improve the quality of short sentence synthesis, experiments are conducted in the variance adapter, and by evaluating the loss calculation of different phoneme sequence lengths, two predictors with different structures are used to predict the pitch, energy and duration of phoneme sequences of different lengths.

[0006] To achieve the above purpose, the technical solution adopted by the present invention is as follows: A zero-sample speech cloning method based on dynamic neural network and feature modulation, the method comprising the following steps: The speaker style encoder extracts the speaker style vector from the reference audio; Perform feature modulation on the speaker style vector based on the SGF algorithm, and input the modulated speaker style vector into the generator; The generator synthesizes target speaker audio based on a dynamic neural network.

[0007] Furthermore, the speaker style encoder includes a Mel style encoder and an MFCC style encoder, which are used to extract speaker characteristic style vectors contained in the Mel spectrogram and the Mel frequency cepstral coefficient respectively, and concatenate the extracted speaker characteristic style vectors to obtain a final speaker style vector.

[0008] Further, in the speaker style encoder; The MFCC style encoder uses a bidirectional LSTM to process the MFCC data to extract the temporal changes of the MFCC features and the dependencies in the sequence; performs self-attention calculations on the extracted MFCC features; averages the output of the self-attention mechanism along the time axis to obtain a one-dimensional feature style vector with the same dimension as the output of the Mel style encoder; The one-dimensional style vector output by the Mel style encoder is concatenated with the one-dimensional feature style vector output by the MFCC style encoder to obtain the final style vector.

[0009] Furthermore, the feature modulation of the speaker style vector based on the SGF algorithm includes: set up represents the feature vector of phoneme sequence, where Represents the dimension of the phoneme sequence feature vector. To standardize: , , , In the formula, represents the normalized feature vector, represents the mean of the eigenvector, represents the variance of the eigenvector, represents the feature vector index, is the dimension of the feature vector; The final style vector is modulated by four parameters: , , represents the first scaling factor obtained by processing the final style vector through the hyperbolic tangent function; represents the bias factor obtained by processing the final style vector through the hyperbolic tangent function; Represents the baseline feature obtained by processing the final style vector through the hyperbolic tangent function, which is used to balance and ; Represents the dynamic adjustment factor obtained by processing the final style vector through the Sigmoid function; Four parameters are applied to the normalized feature vectors via a weighting operation: , , , In the formula, Represents the feature sequence after fusion of style vector.

[0010] Furthermore, the generator includes a phoneme encoder, a variance adapter and a mel-spectrogram decoder, wherein the phoneme encoder is used to convert a phoneme embedding sequence into a phoneme hidden sequence containing the speaker's personalized speech style characteristics; the variance adapter is used to add pitch and energy information to the phoneme hidden sequence, and adjust the length of the phoneme hidden sequence according to the predicted duration of each phoneme; the mel-spectrogram decoder is used to inject the style vector into the adjusted phoneme hidden sequence and convert it into a mel-spectrogram sequence.

[0011] Furthermore, the variance adapter includes different predictors corresponding to long phoneme sequences and corresponding to short phoneme sequences.

[0012] Furthermore, the predictor architecture for long phoneme sequences includes two layers of one-dimensional convolutional networks with ReLU activation, each network is followed by a normalization and dropout layer, and a linear layer is connected after the last normalization and dropout layer; the predictor architecture for short phoneme sequences includes two layers of one-dimensional convolutional networks with ReLU activation, each network is followed by a normalization and dropout layer, and a one-dimensional convolutional network is connected after the last normalization and dropout layer.

[0013] Furthermore, by minimizing the Mel-spectrogram reconstruction loss , duration prediction loss , Energy prediction loss , pitch prediction loss Optimize the network model consisting of the speaker encoder and generator.

[0014] The beneficial effects of the present invention are: The present invention proposes a multi-speaker adaptive TTS model with high similarity and generalization ability: the dynamic style speech synthesis model (DS-TTS), which can perform speech cloning in zero-sample mode; proposes an innovative dual-style encoder architecture and Style Gating-Film to filter out information that has an adverse effect on learning during the training process, significantly improving the ability to capture and express the speaker's style; proposes a dynamic neural network method based on sample length to more flexibly adapt to changes in input, thereby performing better when processing texts of different lengths. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is an overall framework diagram of the zero-sample speech cloning method based on dynamic neural network and feature modulation of the present invention; Figure 2 This is a detailed diagram of the network components of the present invention. DETAILED DESCRIPTION

[0016] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0017] As mentioned above, with the rapid development of text-to-speech (TTS) models, people have higher and higher requirements for personalized audio generation. As a subtask of TTS, the goal of the zero-shot voice cloning task is to synthesize the audio of the target speaker without seeing the speaker given a single audio and arbitrary text. Although existing voice cloning models can clone the voices of unseen speakers, they still produce some unnatural and poorly similar audio. In order to enhance the style attributes of unseen speakers and the generalization ability of the model, the present invention introduces the DS-TTS model, which uses two style encoders to extract the speaker style vector as conditional information, which is then input into the generator through Style Gating-Film (SGF). At the same time, in order to solve the problem of poor synthesis effect of short sentences, the present invention designs a dynamic neural network in the variance adapter, uses a linear layer structure to ensure the effective processing of long phoneme sequences, and uses a convolutional layer structure to improve the synthesis effect of short phoneme sequences. Experiments on the VCTK dataset show that the model proposed in the present invention performs well on the zero-shot voice cloning task compared with the current state-of-the-art models.

[0018] like Figure 1 As shown, the overall framework of the zero-sample speech cloning method based on dynamic neural network and feature modulation of the present invention, StyleSpeech is used as the basic framework of the model, and the model is mainly composed of a speaker style encoder and a generator, wherein the Mel style encoder and the MFCC style encoder extract the speaker style vector from the audio, and the generator fuses the style vector to the TTS architecture through Style Gating-Film (SGF) to synthesize the target speaker audio. In this process, how to inject the speaker's speaking style into the TTS architecture is very important. In addition, the present invention also introduces a dynamic neural network as an innovation point to improve the speaker generalization ability in short text synthesis. The dynamic neural network can dynamically select different paths or processing modes according to the input features or conditions. Through the above method, the specific speech features of each speaker can be captured more accurately, thereby improving the naturalness of the synthesized audio and the similarity of the speaker.

[0019] refer to Figure 2 As shown, the various parts of the network framework of the present invention are introduced below.

[0020] 1. Feature modulation; In the present invention, the audio is converted into a mel-spectrogram and MFCC as input, the speaker style is obtained through the mel-style encoder and the MFCC style encoder, and then the speaker's final style vector is obtained by splicing. The speaker's final style vector can capture the speaker's personalized features such as pitch, speaking speed, intonation, pronunciation habits, etc. Specifically, the mel-style encoder extracts the speaker features in the mel-spectrogram, and the MFCC style encoder extracts the speaker features in the mel-frequency cepstral coefficients (MFCC).

[0021] The style encoder designed by the present invention includes the following three parts: LSTM: Use bidirectional LSTM to process MFCC data to capture the temporal changes of MFCC and the dependencies in the sequence.

[0022] Multi-head self-attention mechanism: Perform self-attention calculation on the extracted MFCC features to better understand the global context information.

[0023] Average pooling: The output of the self-attention mechanism is averaged along the time axis to obtain a one-dimensional style vector with the same dimension as the output of the Mel-style encoder. Finally, the outputs of the two are concatenated together to obtain the final style vector.

[0024] The final style vector will be input into the feed-forward transformer block (FFT block) of the phoneme encoder and mel-spectrogram decoder respectively. The FFT block is an important network architecture in the phoneme encoder and mel-spectrogram decoder.

[0025] The generator consists of three parts: a phoneme encoder, a variance adapter, and a mel-spectrogram decoder. The phoneme encoder converts a phoneme embedding sequence into a phoneme hidden sequence that contains the speaker's personalized speech style characteristics. The variance adapter is able to predict the pitch, energy, and duration of each phoneme. It adds the pitch and energy information to the phoneme hidden sequence, and then adjusts the length of the phoneme hidden sequence according to the predicted duration of each phoneme. Finally, after the mel-spectrogram decoder injects the final style vector into the adjusted phoneme hidden sequence, it is converted into a mel-spectrogram sequence.

[0026] Typically, the style vector can be fused with feature information through conditional generation methods such as concatenation, direct addition, and Style-Adaptive Layer Norm to obtain a hidden sequence of phonemes that contains the speaker's personalized speech style features. Among them, the concatenation and direct addition methods may cause confusion of semantic information and fail to accurately convey subtle changes in speaking style. Although Style-Adaptive Layer Norm can adaptively scale and offset feature vectors according to the style vector, it cannot filter out information that may have a negative impact on learning. Therefore, the present invention improves the Style-Adaptive Layer Norm method and proposes Style Gating-FiLM (SGF), which accepts the final style vector w and modulates the feature information. SGF introduces an intermediate parameter based on FiLM. and , filtering out some information that has a negative impact on learning, so that the model can more finely control the mean and variance. represents the feature vector of phoneme sequence, where is the dimension of the feature vector. First, the feature vector of the phoneme sequence To standardize:

[0027] , , , is the eigenvector, is the normalized eigenvector, is the mean of the eigenvectors, is the variance of the eigenvector, represents the feature vector index, is the dimension of the feature vector. Represents the feature vector Middle elements, ranging from 1 to N.

[0028] The final style vector is then modulated by four parameters: , , , represents the final style vector processed by the hyperbolic tangent function , get the first scaling factor.

[0029] Represents the final style vector processed by the hyperbolic tangent function, which has the same effect as Similarly, what can be called a bias factor is used to provide flexibility in feature modulation, ensuring diversity in the final result.

[0030] , represents the final style vector processed by the hyperbolic tangent function, and obtains the benchmark feature, which is used to provide an alternative to ensure that there is a consistent feature representation during the adjustment process, which can balance and .

[0031] , represents the final style vector processed by the Sigmoid function, with an output value in [0,1], which serves as a dynamic adjustment factor to control , and The combination ratio of can realize adaptive adjustment of the feature vector and affect the overall modulation effect.

[0032] Finally, the modulated parameters are applied to the standardized feature vector through a weighted operation to achieve stylized control of the input feature vector: , , The above two formulas give and It can increase the flexibility of adjusting the style vector, so that the same style vector w can adjust the feature vector More delicate control; , Represents the feature sequence after fusion of style vector.

[0033] This mechanism aims to and Scaling and offsetting normalized features enable adaptive adjustment of features. and Additional modulation flexibility is provided, enabling the modulation process to balance two different parameters, thereby achieving more fine-grained semantic representation and style transfer.

[0034] 2. Dynamic neural network based on the length of phoneme sequence; Although the current model can effectively synthesize the text content of most target speakers, it performs poorly when processing short texts during testing. To address this challenge, the present invention proposes a dynamic neural network based on the length of phoneme sequences, and designs different predictors in the variance adapter to deal with short phoneme sequences and long phoneme sequences, so as to reasonably allocate computing resources according to the length of the input phoneme sequence. This dynamic architecture not only reduces the redundant calculation of simple samples, but also maintains its representation ability when identifying complex samples. With this strategy, the voice cloning results can be optimized more effectively, especially when processing short texts. This method not only improves the quality of synthesized speech, but also enhances the model's ability to capture the speech characteristics of the target speaker.

[0035] In the present invention, sentences with a phoneme sequence length less than or equal to 85 are classified as short phoneme sequences, and sentences with a phoneme sequence length greater than 85 are classified as long phoneme sequences. In the experiment, the MSE loss is used to calculate the predicted values ​​of pitch, energy, and duration. With the true value The difference between the two is that the original predictor of the basic framework StyleSpeech performs well when cloning long phoneme sequences. Therefore, the present invention specifically designs a pitch, energy and duration predictor framework for short phoneme sequences: ; is the i-th characteristic component of the predicted value; is the i-th eigenvalue of the true value, and N is the dimension of the eigenvector; Represents the predicted value of pitch, energy, and duration and the true value The mean square error of .

[0036] In the original predictor, the model uses a linear layer for projection. It has a strong fitting ability and can learn more complex mapping relationships, thus better adapting to complex data structures. However, in short sentence synthesis, using a linear layer for projection may lead to overfitting problems of the model. Therefore, the present invention adopts a convolutional layer with local connection and weight sharing features to better adapt to the characteristics of short sentences. Finally, the modified variance adapter architecture is as follows Figure 2 As shown in Figure 2, the predictor architecture for pitch, energy, and duration for long phoneme sequences consists of two layers of 1D convolutional networks with ReLU activations, each followed by a normalization and dropout layer, and a linear layer connected after the last normalization and dropout layer. The predictor architecture for short phoneme sequences consists of two layers of 1D convolutional networks with ReLU activations, each followed by a normalization and dropout layer, and a 1D convolutional network connected after the last normalization and dropout layer.

[0037] During training, the Mel-spectrogram reconstruction loss is minimized , duration prediction loss , Energy prediction loss , pitch prediction loss To optimize the model, the Mel-spectrogram reconstruction loss The MAE loss function is used, and the other losses use the MSE loss function.

[0038] The total loss is as follows: .

[0039] During the training phase, the present invention uses the clean and noise-free train-clean-100 and train-clean-360 datasets from the LibriTTS dataset, which contain data from 1,148 speakers. In addition, the VCTK dataset is used to evaluate unseen speakers, which contains 108 speakers.

[0040] Before training, some preprocessing is required for the audio and text data: 1) The sampling rate of the training and evaluation data is converted to 16kHz; 2) The spectrogram is extracted using an FFT size of 1024, a jump size of 256, and a window size of 1024 samples, and converted into a Mel-spectrogram with 80 frequency bins; 3) 20 MFCC coefficients are extracted from the Mel-spectrogram as features using librosa, and the MFCC features are normalized using the L2 norm; 4) The audio data and text data are aligned using Montreal ForcedAligner to obtain the duration of each phoneme.

[0041] Melgan is used as the vocoder. The outputs of the Mel style encoder and the MFCC style encoder are both 128-dimensional vectors, which are then concatenated into a 256-dimensional style vector. The Style Gating-FiLM layer is placed in the FFT of the phoneme encoder and the Mel spectrogram decoder. In the initialization phase, the parameters The bias is set to 1.0, and the other three parameters are , and The bias of is set to 0. This design is intended to ensure that the model has a certain balance in the initial stage, so as to better adapt to the subsequent style modulation.

[0042] This method sets the model training batch size to 24 and uses the Adam optimizer with the following parameters: , , During the training, 200,000 steps were trained to obtain the final optimized model for practical use.

[0043] Audio samples of 108 different speakers are randomly selected from the VCTK dataset as a test set to evaluate the generalization ability of the model to unseen speakers. The model is compared with audio samples generated by several baselines, and the results of WER and SMCS are shown in Table 1. Obviously, DS-TTS performs well. Although the WER of DS-TTS is slightly higher than StyleTTS2 and XTTS v2, it is significantly better than them in terms of similarity. The SMCS performance of VALL-EX is similar to that of DS-TTS, but its WER is 0.228, which is relatively poor. This may be due to the limited amount of training data used by the author, only 704 hours of English training data. The original English ASR data of LibriLight contains about 60,000 hours of unlabeled speech, which is obviously richer. Overall, the model of the present invention performs well in speaker similarity performance, while having low word error rate and high-quality speech synthesis.

[0044] Table 1

[0045]

[0046] Among them, the baseline models involved in the table are introduced as follows: GT: stands for True Voice.

[0047] StyleSpeech: Uses a Mel-style encoder to extract the speaker’s style and incorporates it into a speech synthesis model based on FastSpeech2 through the Style-AdaptiveLayer Norm to synthesize the target speaker’s audio.

[0048] Retrained StyleSpeech: To ensure the accuracy of normal training and comparison of the model, the batch size was adjusted to 24, the model was trained for 200,000 steps, and the StyleSpeech model was retrained using the same dataset partitioning.

[0049] YourTTS: A zero-shot speaker-adapted speech cloning model based on VITS, using speaker consistency loss as the loss function and evaluated using checkpoints in Coqui TTS.

[0050] VALL-EX: A cross-lingual neural encoding language model based on VALL-E that uses reference audio and text as cues to predict the acoustic symbol sequence of target audio.

[0051] StyleTTS2: A voice cloning model that uses style diffusion and adversarial training with a large speech language model to achieve high-quality TTS synthesis.

[0052] Coqui AI XTTS v2: A zero-shot text-to-speech model based on Tortoise that uses the Perceiver model to add prefixes to the GPT decoder to capture speaker characteristics.

[0053] DS-TTS: A zero-shot speech cloning model proposed in this paper generates highly similar target audio from reference audio through feature modulation and dynamic neural network.

[0054] Subjective and objective evaluation metrics are used to assess the naturalness and similarity of synthesized target speaker utterances.

[0055] 1) Subjective indicators: In terms of subjective evaluation, the mean opinion score and similarity mean opinion score are used to evaluate the quality of audio synthesis. The evaluation range of these indicators is 1 to 5, with 5 being the best performance.

[0056] Mean Opinion Score (MOS): Use MOS to evaluate the clarity of audio and the fluency of sentences, focusing on the presence of background noise and the naturalness and fluidity of prosody, including intonation, pitch, and rhythm.

[0057] Similarity Mean Opinion Score (SMOS): Use SMOS to evaluate people’s subjective perception of the similarity between the synthesized audio and the reference audio.

[0058] 2) Objective indicators: In terms of objective evaluation, word error rate and speaker embedding cosine similarity are used to evaluate the quality of audio synthesis.

[0059] Word Error Rate (WER): WER is used to measure the difference between the reference text and the text recognized by the ASR model. The lower the WER value, the more accurate the pronunciation of the synthesized audio. The present invention selects the basic model of the whisper library as the ASR model.

[0060] Speaker Embedding Cosine Similarity (SMCS): SMCS is used to compare the cosine similarity between the synthesized audio and the real audio. The higher the SMCS value, the higher the similarity of the speaker's voice features in the synthesized audio. The present invention selects Resemblyzer to extract the speaker representation.

[0061] To further evaluate the performance of the model, the effects of different baseline models on short and long phoneme sequences were tested. The results of objective and subjective evaluations are shown in Table 2. First, the comparison of long and short phoneme sequences shows that the DS-TTS model has good adaptability when processing reference speech of any length. Second, through the results of objective indicators, it was found that DS-TTS achieved the best balance of WER and SMCS among all models. Finally, the subjective evaluation results show that under the same small batch data training conditions, the audio generated by DS-TTS is better than YourTTS, StyleSpeech, and Retrained StyleSpeech in terms of MOS. However, compared with StyleTTS2 and XTTS v2, the effect of the model may be slightly inferior. This difference is understandable because StyleTTS2 and XTTS v2 use extremely large data sets, and the training time of both is much longer than the settings used in the present invention. Most importantly, in terms of subjective similarity, DS-TTS shows strong generalization ability, and the generated audio is highly similar to the audio of the target speaker.

[0062] Table 2

[0063]

[0064] In order to evaluate the performance of different genders and English accents, the present invention selected 100 audio samples for each case as the test set. As shown in Table 3, the model of the present invention performs well in generating speech similar to the original speech, with no obvious differences between male and female speakers, while accurately capturing the characteristics of various accents. This further verifies that the features adopted in this paper can effectively capture the commonalities and differences between male and female and various English accents, allowing the model to maintain balanced performance during the generation process.

[0065] Table 3

[0066]

[0067] Appropriate threshold selection can improve the performance of the model and make it perform better in the voice cloning task. According to the ablation experiment results in Table 4, the present invention evaluates multiple models with thresholds between 75 and 95. Among them, the model with a threshold of 85 performed best in the test. At a threshold of 85, the model has a significant improvement in similarity and the lowest pronunciation recognition error rate. This may be because the threshold of 85 happens to be a certain balance point in the dynamic neural network, which can effectively process phoneme sequences of various lengths without prematurely truncating important information or increasing the computational burden of processing too long sequences.

[0068] Table 4

[0069]

[0070] In addition, the effectiveness of the model components is studied as shown in Table 5. Removing MFCC features significantly reduces the quality of the generated speech. In addition, using only long predictors or short predictors significantly reduces the model performance. The results show that MFCC features play an important role in feature modulation, and the dynamic neural network further improves the adaptive performance of the model.

[0071] Table 5

[0072]

[0073] In summary, the present invention proposes DS-TTS, a text-to-speech model for zero-shot speech cloning, a zero-shot speech cloning method based on dynamic neural networks and feature modulation. In order to enhance the generalization ability of the model, Mel-style encoder and MFCC style encoder are used to extract the speaker's style vector, and Style Gating-Film (SGF) is used to feature modulate the feature vector. At the same time, in order to solve the problem of poor synthesis of short phoneme sequences, the input sequence is classified according to the sample size, and a dynamic neural network is used in the variance adapter to more accurately predict the speaker's pitch, energy and duration of short and long samples. Experimental results show that DS-TTS outperforms existing methods in the comprehensive performance of multiple indicators.

[0074] The present invention can achieve accurate and natural speech generation with very little audio data of the target speaker through deep neural network optimization. This technology not only makes breakthroughs in the natural fluency of speech synthesis, but also correspondingly enhances the anti-tampering and anti-counterfeiting capabilities of the generation model by optimizing the generation process and the dynamic deployment of the neural network. This anti-tampering capability can effectively induce the alarm of voice in information dissemination, ensuring that the generated voice will not be maliciously tampered with or used for malicious information dissemination in information dissemination channels such as social media and news reports. The present invention can provide a higher security speech generation technology, reduce the generation and dissemination of false voices, thereby reducing the impact of false information on public cognition, and ensuring the authenticity of information and the stability of socialization.

[0075] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A zero-sample speech cloning method based on dynamic neural network and feature modulation, characterized in that: The method comprises the following steps: The speaker style encoder extracts the speaker style vector from the reference audio; Perform feature modulation on the speaker style vector based on the SGF algorithm, and input the modulated speaker style vector into the generator; The generator synthesizes target speaker audio based on a dynamic neural network.

2. The zero-sample speech cloning method based on dynamic neural network and feature modulation according to claim 1, characterized in that: The speaker style encoder includes a Mel style encoder and an MFCC style encoder, which are used to extract speaker feature style vectors contained in the Mel spectrogram and the Mel frequency cepstral coefficient respectively, and concatenate the extracted speaker feature style vectors to obtain a final speaker style vector.

3. The zero-sample speech cloning method based on dynamic neural network and feature modulation according to claim 2, characterized in that: In the speaker style encoder; The MFCC style encoder uses a bidirectional LSTM to process the MFCC data to extract the temporal changes of the MFCC features and the dependencies in the sequence; performs self-attention calculations on the extracted MFCC features; averages the output of the self-attention mechanism along the time axis to obtain a one-dimensional feature style vector with the same dimension as the output of the Mel style encoder; The one-dimensional style vector output by the Mel style encoder is concatenated with the one-dimensional feature style vector output by the MFCC style encoder to obtain the final style vector.

4. The zero-sample speech cloning method based on dynamic neural network and feature modulation according to claim 1, characterized in that: The feature modulation of the speaker style vector based on the SGF algorithm includes: set up represents the feature vector of phoneme sequence, where Represents the dimension of the phoneme sequence feature vector. To standardize: ， ， ， In the formula, represents the normalized feature vector, represents the mean of the eigenvector, represents the variance of the eigenvector, represents the feature vector index, is the dimension of the feature vector; The final style vector is modulated by four parameters: ， ， represents the first scaling factor obtained by processing the final style vector through the hyperbolic tangent function; represents the bias factor obtained by processing the final style vector through the hyperbolic tangent function; Represents the baseline feature obtained by processing the final style vector through the hyperbolic tangent function, which is used to balance and ; Represents the dynamic adjustment factor obtained by processing the final style vector through the Sigmoid function; Four parameters are applied to the normalized feature vectors via a weighting operation: ， ， ， In the formula, Represents the feature sequence after fusion of style vector.

5. The zero-sample speech cloning method based on dynamic neural network and feature modulation according to claim 1, characterized in that: The generator includes a phoneme encoder, a variance adapter and a mel-spectrogram decoder. The phoneme encoder is used to convert a phoneme embedding sequence into a phoneme hidden sequence containing the speaker's personalized voice style characteristics; the variance adapter is used to add pitch and energy information to the phoneme hidden sequence, and adjust the length of the phoneme hidden sequence according to the predicted duration of each phoneme; the mel-spectrogram decoder is used to inject the style vector into the adjusted phoneme hidden sequence and convert it into a mel-spectrogram sequence.

6. The zero-sample speech cloning method based on dynamic neural network and feature modulation according to claim 1, characterized in that: The variance adapter includes different predictors for long phoneme sequences and for short phoneme sequences.

7. The zero-sample speech cloning method based on dynamic neural network and feature modulation according to claim 6, characterized in that: The predictor architecture for long phoneme sequences consists of two layers of one-dimensional convolutional networks with ReLU activations, each followed by a normalization and dropout layer, and a linear layer connected after the last normalization and dropout layer; the predictor architecture for short phoneme sequences consists of two layers of one-dimensional convolutional networks with ReLU activations, each followed by a normalization and dropout layer, and a one-dimensional convolutional network connected after the last normalization and dropout layer.

8. The zero-sample speech cloning method based on dynamic neural network and feature modulation according to claim 1, characterized in that: By minimizing the Mel-spectrogram reconstruction loss , duration prediction loss , Energy prediction loss , pitch prediction loss Optimize the network model consisting of the speaker encoder and generator.

Images