Tone clone conversion method based on GPT-SoVITS model

Abstract

The invention discloses a tone clone conversion method based on a GPT-SoVITS model. The method comprises the following steps: S1, data preparation; preparing voice data with the same timbre, wherein the voice data comprises original voice and corresponding text content; the voice data are used for training a GPT-SoVITS model; s2, data processing; carrying out slicing processing on the voice data file, and carrying out text data annotation on the voice slicing file; s3, model training; using the prepared data to train a GPT-SoVITS model; in the training process, the GPT-SoVITS model learns how to convert text contents into voices with corresponding timbre features; s4, voice generation; after the GPT-SoVITS model is trained, any text or voice content is input, and the GPT-SoVITS model generates corresponding voice according to the text or voice content. According to the invention, through a smaller number of voice material libraries, higher-quality and more vivid voice is cloned and generated.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a timbre cloning conversion method based on a GPT-SoVITS model. Background Art

[0002] The most common audio cloning conversion technologies currently use two solutions: RVC and SoVITS. However, both technologies have many limitations.

[0003] RVC (Retrieval-based Voice Conversion): This method usually involves using a large-scale voice database or voice library to retrieve sound samples similar to the input voice, and converting the input voice into a voice similar to the retrieved sound sample, using the retrieved voice as the target for timbre conversion.

[0004] It is usually applied to voice conversion tasks, such as converting one person's voice into another person's voice. It is commonly used in dialogue scenarios and performs poorly in singing conversion.

[0005] SoVITS (SoftVC VITS Singing Voice Conversion): This is a neural network-based voice conversion method, which usually uses a variational autoencoder (VAE) or other neural network architectures, focusing on singing voice conversion. Its goal is to convert the input singing voice sample into a timbre with different characteristics, such as gender, pitch, etc. Common application scenarios are singing voice conversion, such as converting a male voice into a female voice, changing the voice pitch or other features, but the effect is poor in voice generation in dialogue scenarios.

[0006] Therefore, the prior art has defects and needs to be improved. Summary of the invention

[0007] The technical problem to be solved by the present invention is to provide a timbre cloning conversion method based on the GPT-SoVITS model, which can clone and generate higher quality and more realistic voices through a small amount of voice material library.

[0008] The technical solution of the present invention is as follows: a timbre cloning conversion method based on the GPT-SoVITS model is provided, comprising the following steps.

[0009] S1: Data preparation: prepare speech data with the same timbre, including the original speech and the corresponding text content; these speech data are used to train the GPT-SoVITS model.

[0010] S2: Data processing: Slice the speech data file and annotate the speech slice file with text data.

[0011] S3: Model training: Use the prepared data to train the GPT-SoVITS model. During the training process, the GPT-SoVITS model learns how to convert text content into speech with corresponding timbre features.

[0012] S4: Speech generation: After the GPT-SoVITS model is trained, by inputting any text or speech content, the GPT-SoVITS model generates the corresponding speech based on the text or speech content.

[0013] Furthermore, the GPT-SoVITS model used for training adds a residual quantization layer to the original SOVITS input. The input of this residual quantization layer contains the text features and timbre features of the audio. After training, the GPT-SoVITS model obtains a model that can convert text into residual quantization layer input.

[0014] Furthermore, the input of the encoder of the GPT Sovits model includes two parts: the semantic embedding of the text sequence (semantic token embedding) and the embedding of the phoneme sequence (phoneme token embedding); the GPT Sovits model training process is as follows.

[0015] A1: Semantic embedding and phoneme embedding are added with position information through position encoding (SinePosistionalEmbedding), and then connected together as the input of the encoder.

[0016] A2: The encoder uses the Transformer’s self-attention mechanism to model the input sequence, capture the long-range dependencies between text and phonemes, and outputs the encoded sequence representation y.

[0017] A3: The encoder output y and a target (target) calculate the cross entropy loss (cross_entropy) to complete the training. In the speech synthesis task, the target may be the expected acoustic feature sequence, such as mel spectrum, etc. This encoder aims to fuse the text semantics and phoneme information to generate a representation y that can well predict the target acoustic features.

[0018] Furthermore, the training also includes: the encoder and a vocoder model are trained together; the vocoder receives the y output by the encoder and converts it into a final speech waveform.

[0019] Furthermore, the training process is to minimize the difference between the encoder output y and the actual acoustic feature sequence (reconstruction loss), so that the entire model can generate high-quality speech from text and phonemes; after the training is completed, only the encoder part is retained for inference, the text and speaker ID are input, the predicted acoustic feature sequence is output, and then synthesized into speech through the vocoder.

[0020] Furthermore, semantic information is the result of using the hubert autoencoder information SSL of the audio to enter the residual quantization layer output of sovits; the semantic information includes text and timbre features; among them, phoneme features and berf features are phoneme features for text, similar to pinyin.

[0021] Furthermore, the reasoning process is as follows.

[0022] B1: One input is a prompt text, whose semantic information will be extracted and converted into a semantic embedding vector representation through TokenEmbedding; the other input is the speaker's phoneme sequence, which is also converted into a phoneme embedding vector representation through TokenEmbedding. Token Embeddings is a method of converting words in text into vector representations.

[0023] B2: Semantic embedding and phoneme embedding are respectively added with position information through the position encoding (SinePositionalEmbedding) module.

[0024] B3: The encoded semantic embedding and phoneme embedding are concatenated and input into the Transformer Encoder module; xy_attn_mask is the mask vector that controls the scope of attention calculation.

[0025] B4: The encoder (Transformer Encoder) encodes the concatenated input sequence through the self-attention mechanism, captures the dependencies between different modalities, and outputs the encoded sequence representation y; the different modalities include semantics and phonemes.

[0026] B5: The encoder output y is transformed through a linear layer.

[0027] B6: The obtained y vector is the final output of the GPT Sovits model, which predicts the acoustic feature sequence (possibly mel spectrum, etc.) corresponding to the input prompt text (prompt).

[0028] B7: Input y into an additional vocoder model; the speech waveform corresponding to the input prompt text can be finally synthesized.

[0029] Furthermore, during inference, phoneme and berf use the text features to be generated; semantic information is the encoding features generated by the reference audio; during inference, starting from the reference audio, based on the text features, the semantic information encoding is predicted backwards one by one until the end; the returned result is equivalent to the concatenation of the two segments, which can be directly intercepted.

[0030] Using the above scheme, the present invention provides a voice cloning conversion method based on the GPT-SoVITS model, and GPT-SoVITS combines the two technologies of GPT and SoVITS. GPT is a natural language processing model based on Transformer, which has a strong text generation capability. SoVITS is a voice conversion technology based on deep learning, which can convert one person's voice into another person's voice. By combining these two technologies, GPT-SoVITS can generate highly realistic voices, and the voice content is consistent with the given text content. Such as speech synthesis, speech cloning, speech conversion, etc. Among them, speech synthesis is the most common application scenario. Through GPT-SoVITS technology, we can generate various voices, such as news reports, voice assistants, virtual characters, etc. In addition, GPT-SoVITS technology can also be used for voice cloning, that is, simulating a person's voice for speech generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 is a flow chart of the method of the present invention;

[0032] Figure 2 Flowchart of the training process;

[0033] Figure 3 It is a flowchart of the reasoning process;

[0034] Figure 4 It is the overall framework diagram of the present invention. DETAILED DESCRIPTION

[0035] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0036] See also Figure 1-Figure 4 The present invention provides a timbre cloning conversion method based on the GPT-SoVITS model, comprising the following steps.

[0037] S1: Data preparation: prepare speech data with the same timbre, including the original speech and the corresponding text content; these speech data are used to train the GPT-SoVITS model.

[0038] S2: Data processing: Slice the speech data file and annotate the speech slice file with text data.

[0039] S3: Model training: Use the prepared data to train the GPT-SoVITS model. During the training process, the GPT-SoVITS model learns how to convert text content into speech with corresponding timbre features.

[0040] S4: Speech generation: After the GPT-SoVITS model is trained, by inputting any text or speech content, the GPT-SoVITS model generates the corresponding speech based on the text or speech content.

[0041] The GPT-SoVITS model used for training adds a residual quantization layer to the original SOVITS input. The input of this residual quantization layer contains the text features and timbre features of the audio. After training, the GPT-SoVITS model obtains a model that can convert text into the residual quantization layer input.

[0042] Figure 2 The process of training the encoder (Transformer Encoder) part of the GPT Sovits model. In this embodiment, the input of the encoder of the GPT Sovits model includes two parts: the semantic embedding (semantic token embedding) of the text sequence and the embedding (phoneme token embedding) of the phoneme sequence; the GPT Sovits model training process is as follows.

[0043] A1: Semantic embedding and phoneme embedding are added with position information through position encoding (SinePosistionalEmbedding), and then connected together as the input of the encoder.

[0044] A2: The encoder uses the Transformer’s self-attention mechanism to model the input sequence, capture the long-range dependencies between text and phonemes, and outputs the encoded sequence representation y.

[0045] A3: The encoder output y and a target (target) calculate the cross entropy loss (cross_entropy) to complete the training. In the speech synthesis task, the target may be the expected acoustic feature sequence, such as mel spectrum, etc. This encoder aims to fuse the text semantics and phoneme information to generate a representation y that can well predict the target acoustic features.

[0046] In this embodiment, the training also includes: the encoder and a vocoder model are trained together; the vocoder receives y output by the encoder and converts it into a final speech waveform.

[0047] In this embodiment, the training process is to minimize the difference between the encoder output y and the actual acoustic feature sequence (reconstruction loss), so that the entire model can generate high-quality speech from text and phonemes; after the training is completed, only the encoder part is retained for inference, the text and speaker ID are input, the predicted acoustic feature sequence is output, and then synthesized into speech through the vocoder.

[0048] In this embodiment, semantic information is the result of using the hubert autoencoder information SSL of the audio to enter the residual quantization layer output of sovits; the semantic information includes text and timbre features; among them, phoneme features and berf features are phoneme features for text, similar to pinyin.

[0049] In this example, see Figure 3 , the reasoning process is as follows.

[0050] B1: One input is a prompt text, whose semantic information will be extracted and converted into a semantic embedding vector representation through TokenEmbedding; the other input is the speaker's phoneme sequence, which is also converted into a phoneme embedding vector representation through TokenEmbedding.

[0051] B2: Semantic embedding and phoneme embedding are respectively added with position information through the position encoding (SinePositionalEmbedding) module.

[0052] B3: The encoded semantic embedding and phoneme embedding are concatenated and input into the Transformer Encoder module; xy_attn_mask is the mask vector that controls the scope of attention calculation.

[0053] B4: The encoder (Transformer Encoder) encodes the concatenated input sequence through the self-attention mechanism, captures the dependencies between different modalities, and outputs the encoded sequence representation y; the different modalities include semantics and phonemes.

[0054] B5: The encoder output y is transformed through a linear layer.

[0055] B6: The obtained y vector is the final output of the GPT Sovits model, which predicts the acoustic feature sequence (possibly mel spectrum, etc.) corresponding to the input prompt text (prompt).

[0056] B7: Input y into an additional vocoder model; the speech waveform corresponding to the input prompt text can be finally synthesized.

[0057] This reasoning process demonstrates that GPT Sovits, as an end-to-end speech synthesis model, can generate corresponding speech directly from text prompts and speaker phonemes. Transformer Encoder combines two input modalities and learns the mapping from semantics to acoustic features, enabling it to efficiently perform speech synthesis. This capability comes from two-stage training - pre-training and fine-tuning, which enables the model to master general speech modeling knowledge and task-specific synthesis skills.

[0058] In this embodiment, during inference, phoneme and berf use the text features to be generated; semantic information is the encoding features generated by the reference audio; during inference, starting from the reference audio, based on the text features, the semantic information encoding is predicted backwards one by one until the end; the returned result is equivalent to the concatenation of the two segments, which can be directly intercepted.

[0059] See also Figure 4 , the overall framework is as follows:

[0060] 1. Input processing

[0061] The input is a text prompt to be synthesized, which is processed by the encoder module and mapped to a latent space representation z.

[0062] 2. Probabilistic Modeling

[0063] GPT-Sovits models the probability distribution p(x|z) of the output speech through two main parts:

[0064] Vocoder model g(z|x).

[0065] Transformer Flow Model

[0066] 3. Vocoder

[0067] The vocoder g first quantizes the speech feature x (Vector Quantization) to obtain a discrete representation m_g, and then passes it through the fully connected layer log(S_g) to obtain the log-likelihood log probability.

[0068] The role of the vocoder is to associate the hidden representation z with the speech feature x, providing supervision for the subsequent Transformer flow model.

[0069] 4. Transformer flow model part

[0070] This part includes a Transformer encoder that maps the prior distribution z_p to another hidden representation m_p.

[0071] The log-likelihood log probability is obtained through the fully connected layer log(S_p) for the generation of speech feature x.

[0072] During training, the goal of log(S_p) is to maximize the log-likelihood log(S_g) produced by the vocoder to achieve the goal of speech reconstruction.

[0073] 5. Reasoning process

[0074] During inference, we start with the prior noise z_p and generate m_p through the Transformer flow.

[0075] Then m_p is input into a decoder network to autoregressively generate a speech feature sequence.

[0076] At the same time, auxiliary information such as speaker embedding and phoneme embedding (PPG) will be used to achieve multi-speaker speech synthesis.

[0077] In general, GPT-Sovits has two generation paths: one is to reconstruct speech based on the vocoder to provide supervision; the other is to directly generate speech through the Transformer stream model. The two parts interact with each other, enabling the model to reconstruct speech with high fidelity from information such as text and speakers. This stream model architecture gives it powerful multi-speaker and long audio generation capabilities.

[0078] In summary, the present invention provides a voice cloning conversion method based on the GPT-SoVITS model, and GPT-SoVITS combines the two technologies of GPT and SoVITS. GPT is a natural language processing model based on Transformer, which has a strong text generation capability. SoVITS is a voice conversion technology based on deep learning, which can convert one person's voice into another person's voice. By combining these two technologies, GPT-SoVITS can generate highly realistic voices, and the voice content is consistent with the given text content. Such as speech synthesis, speech cloning, speech conversion, etc. Among them, speech synthesis is the most common application scenario. Through GPT-SoVITS technology, we can generate various voices, such as news reports, voice assistants, virtual characters, etc. In addition, GPT-SoVITS technology can also be used for voice cloning, that is, simulating a certain person's voice for speech generation.

[0079] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A timbre cloning conversion method based on the GPT-SoVITS model, characterized in that: The following steps are involved: S1: Data preparation: prepare speech data with the same timbre, including the original speech and the corresponding text content; these speech data are used to train the GPT-SoVITS model; S2: Data processing: Slice the voice data file and annotate the voice slice file with text data; S3: Model training: Use the prepared data to train the GPT-SoVITS model. During the training process, the GPT-SoVITS model learns how to convert text content into speech with corresponding timbre features. S4: Speech generation: After the GPT-SoVITS model is trained, by inputting any text or speech content, the GPT-SoVITS model generates the corresponding speech based on the text or speech content.

2. The timbre cloning conversion method based on the GPT-SoVITS model according to claim 1, characterized in that: The GPT-SoVITS model used for training adds a residual quantization layer to the original SOVITS input. The input of this residual quantization layer contains the text features and timbre features of the audio. After training, the GPT-SoVITS model obtains a model that can convert text into residual quantization layer input.

3. The timbre cloning conversion method based on the GPT-SoVITS model according to claim 1, characterized in that: The input of the encoder of the GPT Sovits model consists of two parts: the semantic embedding of the text sequence and the embedding of the phoneme sequence; the training process of the GPT Sovits model is as follows: A1: Semantic embedding and phoneme embedding are positionally encoded to add position information and then concatenated together as the input of the encoder; A2: The encoder uses the Transformer’s self-attention mechanism to model the input sequence, capture the long-range dependencies between text and phonemes, and outputs the encoded sequence representation y; A3: The encoder output y is used to calculate the cross entropy loss with a target to complete the training.

4. The timbre cloning conversion method based on the GPT-SoVITS model according to claim 3 is characterized in that: The training also includes: the encoder and a vocoder model are trained together; the vocoder receives the y output by the encoder and converts it into the final speech waveform.

5. The timbre cloning conversion method based on the GPT-SoVITS model according to claim 4 is characterized in that: The training process is to minimize the difference between the encoder output y and the actual acoustic feature sequence, so that the entire model can generate high-quality speech from text and phonemes; after training, only the encoder part is retained for inference, the text and speaker ID are input, the predicted acoustic feature sequence is output, and then synthesized into speech through the vocoder.

6. The timbre cloning conversion method based on the GPT-SoVITS model according to claim 1, characterized in that: The semantic information is the result of using the hubert autoencoder information SSL of the audio to enter the residual quantization layer output of sovits; the semantic information includes text and timbre features; among them, the phoneme feature and berf feature are phoneme features for the text.

7. The timbre cloning conversion method based on the GPT-SoVITS model according to claim 1, characterized in that: The reasoning process is as follows: B1: One input is a prompt text, whose semantic information will be extracted and converted into a semantic embedding vector representation through TokenEmbedding; the other input is the speaker's phoneme sequence, which is also converted into a phoneme embedding vector representation through TokenEmbedding; B2: Semantic embedding and phoneme embedding are added with position information through the position encoding module respectively; B3: The encoded semantic embedding and phoneme embedding are concatenated and input into the encoder module; Where xy_attn_mask is the mask vector that controls the range of attention calculation; B4: The encoder encodes the concatenated input sequence through the self-attention mechanism, captures the dependencies between different modalities, and outputs the encoded sequence representation y; the different modalities include semantics and phonemes. B5: The encoder output y is transformed through a linear layer; B6: The obtained y vector is the final output of the GPT Sovits model, which predicts the acoustic feature sequence corresponding to the input prompt text; B7: Input y into an additional vocoder model; the speech waveform corresponding to the input prompt text can be finally synthesized.

8. The timbre cloning conversion method based on the GPT-SoVITS model according to claim 1, characterized in that: During inference, phoneme and berf use the text features to be generated; Semantic information is the encoding feature generated by the reference audio. During inference, the reference audio is used as the starting point, and based on the text features, the semantic information encoding is predicted backwards one by one until the end. The returned result is equivalent to the concatenation of the two segments, which can be directly intercepted.

Images