Speech cloning model training method, speech cloning method and speech cloning device

Abstract

The invention discloses a voice cloning model training method and device, and a voice cloning method and device, and the method comprises the steps: constructing a text alignment module, a first pitch extraction module, a second pitch extraction module and a rhythm extraction module, and carrying out the training of the modules, obtaining a trained text alignment module, a trained first pitch extraction module, a trained second pitch extraction module and a trained rhythm extraction module; combining the first training data set with a voice style diffusion module, a decoder and a random duration prediction module to construct a voice clone model, and training the voice clone model by adopting the first training data set to obtain a trained voice clone model; and performing fine tuning on the trained voice clone model by adopting the second training data set to obtain a fine-tuned voice clone model, and cloning high-quality voice close to the target tone only by providing reference voice of which the duration is at least the preset duration.

Description

Technical Field

[0001] The present invention relates to the field of speech synthesis, and in particular to a training method for a speech cloning model, a speech cloning method and a speech cloning device. Background Art

[0002] Voice cloning technology is a branch application of speech synthesis technology, which allows new voice content to be generated by using a small number of target speaker's voice samples. Through voice cloning technology, users can enter any text, and then the system will generate an audio whose timbre and intonation are very close to the target speaker's voice. The practicality of this technology is very wide, including but not limited to personalized virtual assistants, dubbing, personal audio content creation and other fields.

[0003] Currently, one of the biggest challenges facing voice cloning technology is how to generate high-quality synthesized speech using fewer target speaker voice samples or in a shorter preparation time. Traditional methods usually rely on adaptive retraining, which is to fine-tune the existing speech synthesis model using the target speaker's voice data.

[0004] Although voice cloning can be achieved by fine-tuning the model, it has several disadvantages. First, it requires a large number of target speaker voice samples, which may be difficult to obtain quickly in practice. Second, this method usually requires a long training time, which delays the use of voice cloning. Finally, traditional adaptive methods usually require creating and adjusting the model for each new speaker separately, which not only increases the workload but also increases the processing complexity. Summary of the invention

[0005] The purpose of this application is to propose a training method for a speech cloning model, a speech cloning method and a device for the above-mentioned technical problems.

[0006] In a first aspect, the present invention provides a method for training a speech cloning model, comprising the following steps:

[0007] Collecting a plurality of single-speaker voice data and a plurality of multi-speaker voice data respectively and preprocessing them respectively to obtain a first training data set and a second training data set, wherein each piece of training data in the first training data set and the second training data set includes voice data, text data and a corresponding speaker ID, wherein the effective duration of the voice data is a preset duration;

[0008] Constructing a text alignment module, a first pitch extraction module, a second pitch extraction module, and a rhythm extraction module and training them respectively to obtain a trained text alignment module, a trained first pitch extraction module, a trained second pitch extraction module, and a trained rhythm extraction module;

[0009] The trained text alignment module, the trained first pitch extraction module, the trained second pitch extraction module and the trained rhythm extraction module are combined with the speech style diffusion module, the decoder and the random duration prediction module to construct a speech cloning model; the speech cloning model is trained with the first training data set to obtain a trained speech cloning model; the trained speech cloning model is fine-tuned with the second training data set to obtain a fine-tuned speech cloning model; in the training and fine-tuning process of the speech cloning model, the first MFCC feature and the second MFCC feature are respectively extracted from the speech data and the reference speech in the first training data set or the second training data set, and the first MFCC feature and the second MFCC feature are respectively input into the trained first pitch extraction module and the trained second pitch extraction module. The output feature of the trained first pitch extraction module and the output feature of the trained second pitch extraction module are input into the speech style diffusion module to obtain the first feature, the first MFCC feature and the phonemes corresponding to the text data in the first training data set or the second training data set are input into the trained text alignment module to obtain the second feature, the phonemes corresponding to the text data in the first training data set or the second training data set are sequentially passed through the trained rhythm extraction module and the random duration prediction module to obtain the third feature, the first feature, the second feature and the third feature are input into the decoder to obtain the cloned speech, the cloned speech and the first MFCC feature are respectively input into the general speech pre-training model for adversarial training, and the parameters of the speech style diffusion module, the decoder and the random duration prediction module are updated by back propagation.

[0010] Preferably, the text alignment module includes a first convolutional layer, a multi-layer convolutional sequence layer, a convolutional normalization layer, a first fully connected layer, an embedding layer, an attention mechanism layer, an LSTM layer and a second fully connected layer connected in sequence, the multi-layer convolutional sequence layer includes a number of convolutional modules connected in sequence, each convolutional module includes a second convolutional layer, a first batch of normalization layers and a first Relu activation function layer connected in sequence, the convolution kernel size of the first convolutional layer and the convolutional normalization layer is 7×7, and the convolution kernel size of the second convolutional layer is 3×3; the training process of the text alignment module includes: preprocessing the speech data in the public speech data set and extracting the corresponding MFCC features, and then inputting them into the text alignment module for training until convergence to obtain a trained text alignment module.

[0011] Preferably, the first pitch extraction module and the second pitch extraction module have the same structure, both of which include a first convolution sequence layer, a second batch normalization layer, a second Relu activation function layer, a first residual block, a second residual block, a third residual block, a first pooling layer, a second convolution sequence layer, a bidirectional LSMT layer and a third fully connected layer connected in sequence, the first convolution sequence layer includes a third convolution layer, a third batch normalization layer, a third Relu activation function layer and a fourth convolution layer connected in sequence, the second convolution sequence layer includes a fifth convolution layer, a fourth batch normalization layer, a fourth Relu activation function layer and a discard layer connected in sequence, the first residual block, the second residual block and the third residual block adopt a residual block structure, the residual block structure includes a preprocessing layer, a third convolution sequence layer and a sixth convolution layer connected in sequence, and the residual block The input features of the structure are added to the output features of the sixth convolutional layer to obtain the output features of the residual block, the preprocessing layer includes the fifth batch normalization layer and the second pooling layer, the third convolution sequence layer includes the seventh convolutional layer, the sixth batch normalization layer and the eighth convolutional layer connected in sequence, the convolution kernel sizes of the third convolutional layer, the fourth convolutional layer, the seventh convolutional layer and the eighth convolutional layer are all 3×3, and the convolution kernel sizes of the fifth convolutional layer and the sixth convolutional layer are 1×1; the training process of the first pitch extraction module and the second pitch extraction module includes: preprocessing the speech data in the public speech data set and extracting the corresponding MFCC features, and then inputting them into the first pitch extraction module and the second pitch extraction module for training respectively until convergence, so as to obtain the trained first pitch extraction module and the trained second pitch extraction module.

[0012] Preferably, the prosody extraction module includes a pre-trained BERT model, a mask predictor, a word predictor and a multiplier, the mask predictor includes a fourth fully connected layer, the word predictor includes a fifth fully connected layer, the output features of the pre-trained BERT model are respectively input into the mask predictor and the word predictor, the output features of the mask predictor and the output features of the word predictor are input into the multiplier to obtain the output features of the prosody extraction module; the training process of the prosody extraction module includes: pre-processing the speech data in the public speech data set and extracting the corresponding MFCC features, and then inputting them into the prosody extraction module for training until convergence to obtain a trained prosody extraction module.

[0013] Preferably, the speech style diffusion module includes a first spectral normalization layer, a residual block sequence layer, a fifth Relu activation function layer, a second spectral normalization layer, an average pooling layer, a sixth Relu activation function layer and a sixth fully connected layer connected in sequence, the residual block sequence layer includes four residual modules connected in sequence, the residual module includes a third spectral normalization layer, a seventh Relu activation function layer, a fourth spectral normalization layer, a ninth convolutional layer, an eighth Relu activation function layer and a fifth spectral normalization layer connected in sequence, and the input features of the residual module are added to the output features of the fifth spectral normalization layer to obtain the output features of the residual module.

[0014] In a second aspect, the present invention provides a training device for a speech cloning model, comprising:

[0015] A data processing module is configured to collect a plurality of single-speaker speech data and a plurality of multi-speaker speech data respectively and perform preprocessing respectively to obtain a first training data set and a second training data set, wherein the speech duration of the speech data in each piece of training data in the first training data set and the second training data set is a preset duration;

[0016] A component construction module is configured to construct a text alignment module, a first pitch extraction module, a second pitch extraction module and a rhythm extraction module and train them respectively to obtain a trained text alignment module, a trained first pitch extraction module, a trained second pitch extraction module and a trained rhythm extraction module;

[0017] The model construction module is configured to construct a voice cloning model by combining a trained text alignment module, a trained first pitch extraction module, a trained second pitch extraction module and a trained rhythm extraction module with a voice style diffusion module, a decoder and a random duration prediction module; train the voice cloning model using a first training data set to obtain a trained voice cloning model; fine-tune the trained voice cloning model using a second training data set to obtain a fine-tuned voice cloning model; in the training and fine-tuning process of the voice cloning model, extract a first MFCC feature and a second MFCC feature from the voice data and the reference voice in the first training data set or the second training data set, respectively, and input the first MFCC feature and the second MFCC feature into the trained first pitch extraction module and the trained second pitch extraction module, respectively. The second pitch extraction module is inputted, the output features of the trained first pitch extraction module and the output features of the trained second pitch extraction module are inputted into the speech style diffusion module to obtain the first feature, the first MFCC feature and the phonemes corresponding to the text data in the first training data set or the second training data set are inputted into the trained text alignment module to obtain the second feature, the phonemes corresponding to the text data in the first training data set or the second training data set are successively passed through the trained rhythm extraction module and the random duration prediction module to obtain the third feature, the first feature, the second feature and the third feature are inputted into the decoder to obtain the cloned speech, the cloned speech and the first MFCC feature are respectively inputted into the general speech pre-training model for adversarial training, and the parameters of the speech style diffusion module, the decoder and the random duration prediction module are updated by back propagation.

[0018] In a third aspect, the present invention provides a voice cloning method, characterized in that a fine-tuned voice cloning model is obtained by training using the voice cloning model training method described in any implementation of the first aspect, comprising the following steps:

[0019] Acquire the speech data of the target speaker and the text data to be cloned, extract the phonemes corresponding to the text data to be cloned, and the effective duration of the speech data is at least a preset duration;

[0020] The speech data of the target speaker is used as the reference speech and the phonemes corresponding to the text data to be cloned are input into the fine-tuned speech cloning model. During the inference process of the fine-tuned speech cloning model, the corresponding MFCC features are extracted from the reference speech. The MFCC features corresponding to the reference speech are sequentially passed through the trained second pitch extraction module and the speech style diffusion module to obtain the fourth feature. The phonemes corresponding to the text data to be cloned are input into the trained text alignment module to obtain the fifth feature. The phonemes corresponding to the text data to be cloned are sequentially passed through the trained rhythm extraction module and the random duration prediction module to obtain the sixth feature. The fourth feature, the fifth feature and the sixth feature are input into the decoder to obtain the cloned speech.

[0021] In a fourth aspect, the present invention provides a voice cloning device, characterized in that a fine-tuned voice cloning model is obtained by training the voice cloning model training method described in any implementation of the first aspect, comprising:

[0022] A data acquisition module is configured to acquire speech data of a target speaker and text data to be cloned, extract phonemes corresponding to the text data to be cloned, and the effective duration of the speech data is at least a preset duration;

[0023] The cloning module is configured to input the speech data of the target speaker as the reference speech and the phonemes corresponding to the text data to be cloned into the fine-tuned speech cloning model. In the inference process of the fine-tuned speech cloning model, the corresponding MFCC features are extracted for the reference speech, the MFCC features corresponding to the reference speech are sequentially passed through the trained second pitch extraction module and the speech style diffusion module to obtain the fourth feature, the phonemes corresponding to the text data to be cloned are input into the trained text alignment module to obtain the fifth feature, the phonemes corresponding to the text data to be cloned are sequentially passed through the trained rhythm extraction module and the random duration prediction module to obtain the sixth feature, the fourth feature, the fifth feature and the sixth feature are input into the decoder to obtain the cloned speech.

[0024] In a fifth aspect, the present invention provides an electronic device comprising one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation manner in the first aspect.

[0025] In a sixth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any implementation manner in the first aspect.

[0026] Compared with the prior art, the present invention has the following beneficial effects:

[0027] (1) The training method of the speech cloning model proposed in the present invention first separately trains the text alignment module, the first pitch extraction module, the second pitch extraction module and the rhythm extraction module in the speech cloning model, thereby reducing the number of parameters updated by back propagation during the training of the speech cloning model, splitting the model for training, and reducing the difficulty of training.

[0028] (2) The training method of the speech cloning model proposed in the present invention adopts a two-stage training process. First, the speech cloning model is trained using a first training data set based on the speech data of a single speaker, and then a second training data set based on the speech data of multiple speakers is used. The training is conducted in stages, so a large number of speech samples of the target speaker are not required, and there is no need to create and adjust the model for each speaker separately. This can effectively reduce the workload and the data demand.

[0029] (3) The fine-tuned speech cloning model trained by the training method of the speech cloning model proposed in the present invention only needs to provide a reference speech with a duration of at least a preset duration to synthesize high-quality cloned speech close to the target timbre. During inference, only a small amount of reference speech is needed to achieve cloning without the need for further training. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0031] Figure 1 A flowchart of a method for training a voice cloning model according to an embodiment of the present application;

[0032] Figure 2 A schematic diagram of the structure of a text alignment module in a speech cloning model of a training method for a speech cloning model according to an embodiment of the present application;

[0033] Figure 3 A schematic diagram of the structure of a first pitch extraction module and a second pitch extraction module in a voice cloning model of a training method for a voice cloning model in an embodiment of the present application;

[0034] Figure 4 A schematic diagram of a residual block structure in a first pitch extraction module and a second pitch extraction module in a method for training a speech cloning model according to an embodiment of the present application;

[0035] Figure 5 A structural schematic diagram of a prosody extraction module in a speech cloning model of a training method for a speech cloning model according to an embodiment of the present application;

[0036] Figure 6 A schematic diagram of the structure of a speech cloning model in a training phase of a speech cloning model training method according to an embodiment of the present application;

[0037] Figure 7 A structural schematic diagram of a voice style diffusion module in a voice cloning model of a training method for a voice cloning model according to an embodiment of the present application;

[0038] Figure 8 A schematic diagram of a residual structure in a speech style diffusion module of a training method for a speech cloning model according to an embodiment of the present application;

[0039] Fig. 9 A schematic diagram of the structure of a speech cloning model in the inference stage of a method for training a speech cloning model according to an embodiment of the present application;

[0040] Fig.10 A schematic diagram of a training device for a speech cloning model according to an embodiment of the present application;

[0041] Fig.11 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0043] Figure 1 A method for training a voice cloning model provided in an embodiment of the present application is shown, comprising the following steps:

[0044] S1, respectively collect a number of single speaker voice data and a number of multi-speaker voice data and pre-process them respectively to obtain a first training data set and a second training data set, each training data in the first training data set and the second training data set includes voice data, text data and its corresponding speaker ID, wherein the effective duration of the voice data is a preset duration.

[0045] Specifically, data collection is first performed. In the embodiments of the present application, the preset duration is 10s as an example. In other embodiments, other durations may also be selected. The voice data of a single speaker is collected. The effective duration of the voice data is about 24 hours. The voice data is divided into a voice data of about 10s. Correct punctuation marks need to be added where there are pauses to obtain a first training data set. The voice data of multiple speakers is collected, a total of about 1,000 people. The effective duration of each person's voice data is 1 hour. A total of 1,000 hours of voice data are divided into a voice data of about 10s. Correct punctuation marks need to be added where there are pauses to obtain a second training data set. Since a complete voice data can be divided into a large number of voice data with a preset duration, a large number of voice samples of the target speaker are not required.

[0046] The segmentation process of single speaker speech data and multi-speaker speech data is as follows:

[0047] The following formula is used to calculate the speech energy of each speech sampling point in single-speaker speech data and multi-speaker speech data:

[0048]

[0049] Among them, RMS represents the speech energy, x i represents the quantized value of the i-th speech sampling point, and N represents the total number of speech sampling points in the single-speaker speech data and the multi-speaker speech data;

[0050] The following formula is used to calculate the speech decibel of each speech sampling point:

[0051]

[0052] Among them, L p Indicates speech decibel, p rms is the RMS of the speech sampling point, p ref The maximum value of speech energy is expressed as a floating point number, and the maximum value is 1.0;

[0053] The speech sampling points where the speech decibel is below the decibel threshold are determined as silence segments, and the single speaker speech data and multi-speaker speech data are segmented at the silence at the end of the silence segment using a preset duration to obtain a number of speech data with an effective duration of the preset duration.

[0054] Each piece of voice data needs to be labeled with the speaker ID and text data in the following format: path|spkid|text, where path represents the storage path of the voice data, spkid represents the speaker ID, and text represents the text data.

[0055] Further, resample the sampling rate of all speech data to 16,000 Hz. In other alternative embodiments, an appropriate sampling rate can also be selected for resampling. Perform preprocessing processes such as data augmentation, noise reduction, separating background music and human voices on all collected speech data.

[0056] Further, it is also necessary to normalize the text data corresponding to the speech data. The process is as follows:

[0057] 1. Text preprocessing:

[0058] Solve problems such as writing errors in text data and homographs in some languages. For pronunciation characters that the algorithm cannot solve temporarily, use similar pronunciation characters for replacement. For example, replace the pronunciation of "嗯" with "恩", and the pronunciation of "呣" with "母".

[0059] 2. Text normalization:

[0060] Mainly solve the reading of special symbols in text data. For example, convert "2kg" to "两千克", and convert "共465篇,约315万字" to "共四百六十五篇,约三百一十五万字".

[0061] 3. Text to phoneme conversion:

[0062] Convert the text data into phonetic notations. For example, convert "中国" to "zhong1 guo2", and convert "good moning" to "g uh d mao r n ih ng". The same applies to other languages.

[0063] Therefore, the training data in the first training data set and the second training data set includes speech data and the phonemes of its corresponding text data. The speech data is also labeled with the corresponding speaker ID and text data. Similarly, in the deployment and inference stage, it is also necessary to normalize the text data to be cloned to improve the accuracy of speech synthesis.

[0064] S2. Construct a text alignment module, a first pitch extraction module, a second pitch extraction module, and a prosody extraction module, and train them respectively to obtain a trained text alignment module, a trained first pitch extraction module, a trained second pitch extraction module, and a trained prosody extraction module.

[0065] In a specific embodiment, the text alignment module includes a first convolutional layer, a multi-layer convolutional sequence layer, a convolutional normalization layer, a first fully connected layer, an embedding layer, an attention mechanism layer, an LSTM layer, and a second fully connected layer connected in sequence. The multi-layer convolutional sequence layer includes a plurality of convolutional modules connected in sequence, each convolutional module includes a second convolutional layer, a first batch of normalization layers, and a first Relu activation function layer connected in sequence. The convolution kernel size of the first convolutional layer and the convolutional normalization layer is 7×7, and the convolution kernel size of the second convolutional layer is 3×3. The training process of the text alignment module includes: preprocessing the speech data in the public speech data set and extracting the corresponding MFCC features, and then inputting them into the text alignment module for training until convergence to obtain a trained text alignment module.

[0066] Specifically, basic components are constructed, and the basic components include a text alignment module, a first pitch extraction module, a second pitch extraction module, and a rhythm extraction module. Figure 2 , the text alignment module includes a first convolutional layer, a multi-layer convolutional sequence layer, a convolutional normalization layer, a first fully connected layer, an embedding layer, an attention mechanism layer, an LSTM layer, and a second fully connected layer. The convolution kernel size of the first convolutional layer is 7×7, the padding is 3, and the stride is 2. In an embodiment of the present application, there are 5 convolutional modules in the multi-layer convolutional sequence layer, and each convolutional module consists of a second convolutional layer, a first batch of normalization layers, and a first Relu activation function layer. The convolution kernel size of the first convolutional layer is 3×3, the padding is 1, the stride is 1, and the mean of the first batch of normalization layers is 0 and the variance is 1. The input dimension of the convolutional normalization layer is 40, the output dimension is 256, the convolution kernel size is 7×7, the padding is 3, and the stride is 2; the convolutional normalization layer can not only improve the network training speed, but also improve the adversarial and robustness of the network. The output dimension of the first fully connected layer is 128, the output dimension of the embedding layer is 256, the dimension of the attention mechanism layer is 512, and the number of heads is 32; the input dimension of the LSTM layer is 512+256, the output dimension is 512, and the output dimension of the second fully connected layer is 512. The public speech datasets LibriTTS and AiShell are used to train the text alignment module. The data preparation refers to step S1. Before the speech data is input into the text alignment module, MFCC features need to be extracted first, and then the text alignment module is trained until convergence. The trained text alignment module is called a trained text alignment module.

[0067] In a specific embodiment, the first pitch extraction module and the second pitch extraction module have the same structure, and both include a first convolution sequence layer, a second batch normalization layer, a second Relu activation function layer, a first residual block, a second residual block, a third residual block, a first pooling layer, a second convolution sequence layer, a bidirectional LSMT layer and a third fully connected layer connected in sequence. The first convolution sequence layer includes a third convolution layer, a third batch normalization layer, a third Relu activation function layer and a fourth convolution layer connected in sequence. The second convolution sequence layer includes a fifth convolution layer, a fourth batch normalization layer, a fourth Relu activation function layer and a discard layer connected in sequence. The first residual block, the second residual block and the third residual block adopt a residual block structure. The residual block structure includes a preprocessing layer, a third convolution sequence layer and a sixth convolution layer connected in sequence. The input features of the difference block structure are added to the output features of the sixth convolutional layer to obtain the output features of the residual block. The preprocessing layer includes the fifth batch normalization layer and the second pooling layer. The third convolution sequence layer includes the seventh convolutional layer, the sixth batch normalization layer and the eighth convolutional layer connected in sequence. The convolution kernel sizes of the third convolutional layer, the fourth convolutional layer, the seventh convolutional layer and the eighth convolutional layer are all 3×3, and the convolution kernel sizes of the fifth convolutional layer and the sixth convolutional layer are 1×1. The training process of the first pitch extraction module and the second pitch extraction module includes: preprocessing the speech data in the public speech data set and extracting the corresponding MFCC features, and then inputting them into the first pitch extraction module and the second pitch extraction module for training respectively until convergence, so as to obtain the trained first pitch extraction module and the trained second pitch extraction module.

[0068] Specifically, refer to Figure 3, the first pitch extraction module and the second pitch extraction module have the same structure, both of which include the first convolution sequence layer, the second batch normalization layer, the second Relu activation function layer, the first residual block, the second residual block, the third residual block, the first pooling layer, the second convolution sequence layer, the bidirectional LSMT layer and the third fully connected layer. The first convolution sequence layer consists of the third convolution layer, the third batch normalization layer, the third Relu activation function layer and the fourth convolution layer. The convolution kernel size of the third and fourth convolution layers is 3×3, the padding is 1, and the number of input channels of the third batch normalization layer is 64; the number of input channels of the first residual block is 64, and the number of output channels is 128; the number of input channels of the second residual block is 128, and the number of output channels is 192; the number of input channels of the third residual block is 192, and the number of output channels is 256; the number of input channels of the first pooling layer is 256; the second convolution sequence layer consists of the fifth convolution layer, the fourth batch normalization layer, the fourth Relu activation function layer and the drop layer. The convolution kernel size of the fifth convolutional layer is 1×1, the number of input channels of the fourth batch normalization layer is 256, and the drop probability of the drop layer is 0.5; the number of input channels of the bidirectional LSTM layer is 512, the dimension of the hidden state is 256, and the drop probability is 0.3; the number of input channels of the third fully connected layer is 512, and the number of output channels is 722. The structures of the first residual block, the second residual block, and the third residual block are the same, all using the residual block structure, the difference is that the input dimension and the output dimension are different. Figure 4 , the residual block structure includes a preprocessing layer, a third convolutional sequence layer, and a sixth convolutional layer. The input features of the residual block structure are added to the output features of the sixth convolutional layer to obtain the output features of the residual block. The preprocessing layer consists of a fifth batch normalization layer and a second pooling layer. The number of input channels of the fifth batch normalization layer is configurable, which are 64, 128, and 192 respectively. The window size of the second pooling layer is 1×2; the third convolutional sequence layer: consists of a seventh convolutional layer, a sixth batch normalization layer, and an eighth convolutional layer. The convolution kernel size of the seventh convolutional layer is 3×3, and the padding is 1. The number of output channels of the batch normalization layer is configurable, which are 128, 192, and 256 respectively. The convolution kernel size of the eighth convolutional layer is 3×3, and the padding is 1. The convolution kernel size of the sixth convolutional layer is 1×1, and the step size is 1. The public speech dataset LibriTTS is used to train the first pitch extraction module and the second pitch extraction module. The data preparation refers to step S1. Before the speech data is input into the first pitch extraction module and the second pitch extraction module, the MFCC features need to be extracted first. Then the first pitch extraction module and the second pitch extraction module are trained until convergence. The trained first pitch extraction module and the trained second pitch extraction module are called the trained first pitch extraction module and the trained second pitch extraction module.

[0069] In a specific embodiment, the prosody extraction module includes a pre-trained BERT model, a mask predictor, a word predictor and a multiplier, the mask predictor includes a fourth fully connected layer, the word predictor includes a fifth fully connected layer, the output features of the pre-trained BERT model are respectively input into the mask predictor and the word predictor, the output features of the mask predictor and the output features of the word predictor are input into the multiplier to obtain the output features of the prosody extraction module; the training process of the prosody extraction module includes: pre-processing the speech data in the public speech data set and extracting the corresponding MFCC features, and then inputting them into the prosody extraction module for training until convergence to obtain a trained prosody extraction module.

[0070] Specifically, refer to Figure 5 The prosody extraction module includes a pre-trained BERT model, a mask predictor, a word predictor, and a multiplier; the pre-trained BERT model is an open source standard model, and its structure is not repeated here. The mask predictor consists of the fourth fully connected layer, with 768 input channels and 178 output channels; the word predictor consists of the fifth fully connected layer, with 768 input channels and 84827 output channels. The prosody extraction module is trained using a public speech dataset, with a total of 10,000 hours of speech data. The data preparation refers to step S1. Before the speech data is input into the prosody extraction module, MFCC features need to be extracted first, and then the prosody extraction module is trained until convergence. The trained prosody extraction module is called a trained prosody extraction module.

[0071] S3, combining the trained text alignment module, the trained first pitch extraction module, the trained second pitch extraction module and the trained rhythm extraction module with the speech style diffusion module, the decoder and the random duration prediction module to construct a speech cloning model, using the first training data set to train the speech cloning model to obtain a trained speech cloning model; using the second training data set to fine-tune the trained speech cloning model to obtain a fine-tuned speech cloning model; in the training and fine-tuning process of the speech cloning model, extracting first MFCC features and second MFCC features from speech data and reference speech in the first training data set or the second training data set, respectively, and inputting the first MFCC features and the second MFCC features into the trained first pitch extraction module and the trained second pitch extraction module, respectively. The output features of the trained first pitch extraction module and the output features of the trained second pitch extraction module are input into the speech style diffusion module to obtain the first feature, the first MFCC feature and the phonemes corresponding to the text data in the first training data set or the second training data set are input into the trained text alignment module to obtain the second feature, the phonemes corresponding to the text data in the first training data set or the second training data set are successively passed through the trained rhythm extraction module and the random duration prediction module to obtain the third feature, the first feature, the second feature and the third feature are input into the decoder to obtain the cloned speech, the cloned speech and the first MFCC feature are respectively input into the general speech pre-training model for adversarial training, and the parameters of the speech style diffusion module, the decoder and the random duration prediction module are updated by back propagation.

[0072] In a specific embodiment, the speech style diffusion module includes a first spectral normalization layer, a residual block sequence layer, a fifth Relu activation function layer, a second spectral normalization layer, an average pooling layer, a sixth Relu activation function layer and a sixth fully connected layer connected in sequence, the residual block sequence layer includes four residual modules connected in sequence, the residual module includes a third spectral normalization layer, a seventh Relu activation function layer, a fourth spectral normalization layer, a ninth convolutional layer, an eighth Relu activation function layer and a fifth spectral normalization layer connected in sequence, and the input features of the residual module are added to the output features of the fifth spectral normalization layer to obtain the output features of the residual module.

[0073] Specifically, to build a voice cloning model, refer to Figure 6The speech cloning model includes a trained text alignment module, a trained first pitch extraction module, a trained second pitch extraction module, a trained rhythm extraction module, a speech style diffusion module, a decoder and a random duration prediction module. The speech data in the first training data set and the second training data set and the reference speech respectively extract the first MFCC feature and the second MFCC feature. The first MFCC feature and the second MFCC feature are respectively input into the trained first pitch extraction module and the trained second pitch extraction module. The trained first pitch extraction module and the trained second pitch extraction module are respectively connected to the speech style diffusion module. The phonemes corresponding to the text data in the first training data set and the second training data set are respectively input into the trained text alignment module and the trained rhythm extraction module. The trained rhythm extraction module is connected to the random duration prediction module. The trained text alignment module, the speech style diffusion module and the random duration prediction module are respectively connected to the decoder, and the decoder outputs the cloned speech.

[0074] For further reference, Figure 7 , the speech style diffusion module includes the first spectral normalization layer, the residual block sequence layer, the fifth Relu activation function layer, the second spectral normalization layer, the average pooling layer, the sixth Relu activation function layer and the sixth fully connected layer. The convolution kernel size of the first spectral normalization layer is 3×3, the step size is 1, the padding is 1, and the number of output channels is 48; the residual block sequence layer consists of 4 layers of residual modules, the number of input channels is 48, and the number of output channels is 384; the convolution kernel size of the second spectral normalization layer is 5×5, the step size is 1, and the number of output channels is 384; the average pooling layer is used to reduce the feature map to 1×1, and the number of input channels of the sixth fully connected layer is 384, and the number of output channels is 48. Reference Figure 8 The residual module includes a third spectral normalization layer, a seventh Relu activation function layer, a fourth spectral normalization layer, a ninth convolution layer, an eighth Relu activation function layer and a fifth spectral normalization layer. The input features of the residual module are added to the output features of the fifth spectral normalization layer to obtain the output features of the residual module. The convolution kernel size of the third spectral normalization layer is 1×1, and the step size is 1; the convolution kernel size of the fourth spectral normalization layer is 3×3, and the step size is 1; the convolution kernel size of the ninth convolution layer is 3×3, and the step size is 1, and the padding is 1; the convolution kernel size of the fifth spectral normalization layer is 5×5, and the step size is 1, and the padding is 1.

[0075] The random duration prediction module uses the Gaussian upsampling method, and the formula is as follows:

[0076]

[0077] Among them, d pred is the predicted duration of the phoneme, l i represents the i-th phoneme t i The end position of ci represents the i-th phoneme t i The central position of c i =(l i +l i-1 ) / 2, l i-1 Indicates the end position of the i-1th phoneme; Indicates the center position of the phoneme c i The normal distribution probability of , n represents the mean, and σ represents the variance.

[0078] The training of the voice cloning model is divided into two stages. The first stage: the voice cloning model is used as the basic model. The voice cloning model consists of a trained text alignment module, a trained first pitch extraction module, a trained second pitch extraction module, a trained rhythm extraction module and other components. The first training data set is used to train the basic model to obtain a trained voice cloning model. At this time, the trained voice cloning model has the basic ability to clone a single speaker. The second stage: fine-tune the trained voice cloning model and use the second training data set to train to obtain a fine-tuned voice cloning model. Since the second training data set uses multi-speaker voices, after fine-tuning, the fine-tuned voice cloning model has the ability to clone multiple speakers, and there is no need to create and adjust the model for each speaker separately. Reference speech is required during the training process. During training, the target speaker's voice or a randomly selected voice of another speaker is used as a reference voice for adversarial training.

[0079] During the training process of the speech cloning model, the first MFCC feature and the cloned speech are respectively input into the Wavlm model, which is an open source general speech pre-training model for adversarial training, and the speech reconstruction loss function is calculated. The speech reconstruction loss function adopts the Minimax loss function, and then back propagation is performed to update the parameters of the speech style diffusion module, decoder, and random duration prediction module. The parameters of the trained text alignment module, the trained first pitch extraction module, the trained second pitch extraction module, and the trained rhythm extraction module are frozen.

[0080] The speech reconstruction loss function is shown as follows:

[0081]

[0082] Among them, G represents the generator and D represents the discriminator Indicates minimization optimization of the generator G. Indicates the maximum optimization of the discriminator D, E x and E t represents the expected value corresponding to the data, x represents the real data sample, and t represents the noise sample. slm(x) is the optimal solution of the discriminator when inputting x, P wot Represents the probability distribution density of real data, P wog Represents the probability distribution density of generated data. When the input x is real data, D slm The value of (x) is close to 1. When the input x is generated data, D slm The value of (x) is close to 0.

[0083] The embodiment of the present application further provides a voice cloning method, which uses the above-mentioned voice cloning model training method to obtain a fine-tuned voice cloning model, comprising the following steps:

[0084] Acquire the speech data of the target speaker and the text data to be cloned, extract the phonemes corresponding to the text data to be cloned, and the effective duration of the speech data is at least a preset duration;

[0085] The speech data of the target speaker is used as the reference speech and the phonemes corresponding to the text data to be cloned are input into the fine-tuned speech cloning model. During the inference process of the fine-tuned speech cloning model, the corresponding MFCC features are extracted from the reference speech. The MFCC features corresponding to the reference speech are sequentially passed through the trained second pitch extraction module and the speech style diffusion module to obtain the fourth feature. The phonemes corresponding to the text data to be cloned are input into the trained text alignment module to obtain the fifth feature. The phonemes corresponding to the text data to be cloned are sequentially passed through the trained rhythm extraction module and the random duration prediction module to obtain the sixth feature. The fourth feature, the fifth feature and the sixth feature are input into the decoder to obtain the cloned speech.

[0086] Specifically, the fine-tuned voice cloning model is deployed to the server. In the inference stage, the input text data is first preprocessed to extract the phonemes corresponding to the text data, and the MFCC features are extracted for the reference voice. The extracted reference voice MFCC features and the phonemes corresponding to the text data are input into the fine-tuned voice cloning model to infer the cloned voice. Therefore, when the embodiment of the present application faces the voice cloning requirements of the target speaker, it is not necessary to fine-tune the model. It only needs to provide a reference voice within 10 seconds to synthesize high-quality voice close to the target timbre.

[0087] The model structure in the inference phase is different from the model structure during training. Fig. 9 , extract MFCC features from the reference speech and pass them through the trained second pitch extraction module and the speech style diffusion module in sequence to obtain the fourth feature, the phonemes corresponding to the text data are input into the trained text alignment module to obtain the fifth feature, the phonemes corresponding to the text data pass through the trained rhythm extraction module and the random duration prediction module in sequence to obtain the sixth feature, and the fourth feature, the fifth feature and the sixth feature are output as the cloned speech after passing through the decoder.

[0088] Further references Fig.10 As an implementation of the methods shown in the above figures, the present application provides an embodiment of a training device for a speech cloning model. Figure 1 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0089] The present application embodiment provides a training device for a voice cloning model, including:

[0090] The data processing module 1 is configured to collect a plurality of single speaker speech data and a plurality of multi-speaker speech data and perform preprocessing respectively to obtain a first training data set and a second training data set, wherein the speech duration of the speech data in each piece of training data in the first training data set and the second training data set is a preset duration;

[0091] A component construction module 2 is configured to construct a text alignment module, a first pitch extraction module, a second pitch extraction module and a rhythm extraction module and train them respectively to obtain a trained text alignment module, a trained first pitch extraction module, a trained second pitch extraction module and a trained rhythm extraction module;

[0092] The model construction module 3 is configured to construct a voice cloning model by combining the trained text alignment module, the trained first pitch extraction module, the trained second pitch extraction module and the trained rhythm extraction module with the speech style diffusion module, the decoder and the random duration prediction module; the voice cloning model is trained using the first training data set to obtain a trained voice cloning model; the trained voice cloning model is fine-tuned using the second training data set to obtain a fine-tuned voice cloning model; in the training and fine-tuning process of the voice cloning model, the first MFCC feature and the second MFCC feature are respectively extracted from the voice data and the reference voice in the first training data set or the second training data set, and the first MFCC feature and the second MFCC feature are respectively input into the trained first pitch extraction module and the trained second pitch extraction module; The second pitch extraction module is inputted, the output features of the trained first pitch extraction module and the output features of the trained second pitch extraction module are inputted into the speech style diffusion module to obtain the first feature, the first MFCC feature and the phonemes corresponding to the text data in the first training data set or the second training data set are inputted into the trained text alignment module to obtain the second feature, the phonemes corresponding to the text data in the first training data set or the second training data set are successively passed through the trained rhythm extraction module and the random duration prediction module to obtain the third feature, the first feature, the second feature and the third feature are inputted into the decoder to obtain the cloned speech, the cloned speech and the first MFCC feature are respectively inputted into the general speech pre-training model for adversarial training, and the parameters of the speech style diffusion module, the decoder and the random duration prediction module are updated by back propagation.

[0093] The embodiment of the present application further provides a voice cloning device, which uses a fine-tuned voice cloning model trained according to the training method of the voice cloning model described above, comprising:

[0094] A data acquisition module is configured to acquire speech data of a target speaker and text data to be cloned, extract phonemes corresponding to the text data to be cloned, and the effective duration of the speech data is at least a preset duration;

[0095] The cloning module is configured to input the speech data of the target speaker as the reference speech and the phonemes corresponding to the text data to be cloned into the fine-tuned speech cloning model. In the inference process of the fine-tuned speech cloning model, the corresponding MFCC features are extracted for the reference speech, the MFCC features corresponding to the reference speech are sequentially passed through the trained second pitch extraction module and the speech style diffusion module to obtain the fourth feature, the phonemes corresponding to the text data to be cloned are input into the trained text alignment module to obtain the fifth feature, the phonemes corresponding to the text data to be cloned are sequentially passed through the trained rhythm extraction module and the random duration prediction module to obtain the sixth feature, the fourth feature, the fifth feature and the sixth feature are input into the decoder to obtain the cloned speech.

[0096] Fig.11 Schematic diagram of the hardware structure of the electronic device provided by the embodiment of the present invention. Fig.11 As shown, the electronic device of this embodiment includes: a processor 1101 and a memory 1102; wherein the memory 1102 is used to store computer-executable instructions; the processor 1101 is used to execute the computer-executable instructions stored in the memory to implement the various steps performed by the electronic device in the above embodiment. For details, please refer to the relevant description in the above method embodiment.

[0097] Optionally, the memory 1102 may be independent or integrated with the processor 1101 .

[0098] When the memory 1102 is independently provided, the electronic device further includes a bus 1103 for connecting the memory 1102 and the processor 1101 .

[0099] The embodiment of the present invention further provides a computer storage medium, in which computer execution instructions are stored. When the processor 1101 executes the computer execution instructions, the above method is implemented.

[0100] An embodiment of the present invention further provides a computer program product, including a computer program. When the computer program is executed by the processor 1101, the above method is implemented.

[0101] In the embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. Another point, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0102] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to implement the solution of this embodiment.

[0103] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, or each module may exist physically separately, or two or more modules may be integrated into one unit. The above-mentioned module-composed unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0104] The above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor 1101 to perform some steps of the methods of various embodiments of the present application.

[0105] It should be understood that the processor 1101 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or the processor 1101 may be any conventional processor 1101, etc. The steps of the method disclosed in the invention may be directly embodied in the execution of the hardware processor 1101, or may be executed by a combination of hardware and software modules in the processor 1101.

[0106] The memory 1102 may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk memory, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a magnetic disk, or an optical disk.

[0107] The bus 1103 may be an Industry Standard Architecture (ISA), a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus 1103 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus 1103 in the drawings of the present application is not limited to only one bus 1103 or one type of bus 1103.

[0108] The above storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0109] An exemplary storage medium is coupled to the processor 1101, so that the processor 1101 can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor 1101. The processor 1101 and the storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor 1101 and the storage medium can also exist as discrete components in an electronic device or a main control device.

[0110] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0111] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for training a voice cloning model, characterized in that: The following steps are involved: Collecting a plurality of single-speaker speech data and a plurality of multi-speaker speech data respectively and preprocessing them respectively to obtain a first training data set and a second training data set, wherein each piece of training data in the first training data set and the second training data set includes speech data, text data and a corresponding speaker ID, wherein the effective duration of the speech data is a preset duration; Constructing a text alignment module, a first pitch extraction module, a second pitch extraction module, and a rhythm extraction module and training them respectively to obtain a trained text alignment module, a trained first pitch extraction module, a trained second pitch extraction module, and a trained rhythm extraction module; The trained text alignment module, the trained first pitch extraction module, the trained second pitch extraction module and the trained prosody extraction module are combined with a speech style diffusion module, a decoder and a random duration prediction module to construct a speech cloning model; the speech cloning model is trained using the first training data set to obtain a trained speech cloning model; the trained speech cloning model is fine-tuned using the second training data set to obtain a fine-tuned speech cloning model; During the training and fine-tuning process of the speech cloning model, the first MFCC feature and the second MFCC feature are respectively extracted from the speech data and the reference speech in the first training data set or the second training data set, the first MFCC feature and the second MFCC feature are respectively input into the trained first pitch extraction module and the trained second pitch extraction module, the output feature of the trained first pitch extraction module and the output feature of the trained second pitch extraction module are input into the speech style diffusion module to obtain the first feature, the first MFCC feature and the phonemes corresponding to the text data in the first training data set or the second training data set are input into the trained text alignment module to obtain the second feature, the phonemes corresponding to the text data in the first training data set or the second training data set are sequentially passed through the trained prosody extraction module and the random duration prediction module to obtain the third feature, the first feature, the second feature and the third feature are input into the decoder to obtain the cloned speech, the cloned speech and the first MFCC feature are respectively input into the general speech pre-training model for adversarial training, and the parameters of the speech style diffusion module, the decoder and the random duration prediction module are updated by back propagation.

2. The method for training a voice cloning model according to claim 1, characterized in that: The text alignment module includes a first convolutional layer, a multi-layer convolutional sequence layer, a convolutional normalization layer, a first fully connected layer, an embedding layer, an attention mechanism layer, an LSTM layer, and a second fully connected layer connected in sequence. The multi-layer convolutional sequence layer includes a plurality of convolutional modules connected in sequence, each convolutional module includes a second convolutional layer, a first batch of normalization layers, and a first Relu activation function layer connected in sequence. The convolution kernel size of the first convolutional layer and the convolutional normalization layer is 7×7, and the convolution kernel size of the second convolutional layer is 3×3. The training process of the text alignment module includes: preprocessing the speech data in the public speech data set and extracting the corresponding MFCC features, and then inputting them into the text alignment module for training until convergence, so as to obtain a trained text alignment module.

3. The method for training a voice cloning model according to claim 1, characterized in that: The first pitch extraction module and the second pitch extraction module have the same structure, and both include a first convolution sequence layer, a second batch normalization layer, a second Relu activation function layer, a first residual block, a second residual block, a third residual block, a first pooling layer, a second convolution sequence layer, a bidirectional LSMT layer and a third fully connected layer connected in sequence, the first convolution sequence layer includes a third convolution layer, a third batch normalization layer, a third Relu activation function layer and a fourth convolution layer connected in sequence, the second convolution sequence layer includes a fifth convolution layer, a fourth batch normalization layer, a fourth Relu activation function layer and a discard layer connected in sequence, the first residual block, The second residual block and the third residual block adopt a residual block structure, the residual block structure includes a preprocessing layer, a third convolution sequence layer and a sixth convolution layer connected in sequence, the input feature of the residual block structure is added to the output feature of the sixth convolution layer to obtain the output feature of the residual block, the preprocessing layer includes a fifth batch normalization layer and a second pooling layer, the third convolution sequence layer includes a seventh convolution layer, a sixth batch normalization layer and an eighth convolution layer connected in sequence, the convolution kernel size of the third convolution layer, the fourth convolution layer, the seventh convolution layer and the eighth convolution layer is 3×3, and the convolution kernel size of the fifth convolution layer and the sixth convolution layer is 1×1; The training process of the first pitch extraction module and the second pitch extraction module includes: preprocessing the speech data in the public speech data set and extracting the corresponding MFCC features, and then inputting them into the first pitch extraction module and the second pitch extraction module for training respectively until convergence, so as to obtain a trained first pitch extraction module and a trained second pitch extraction module.

4. The method for training a voice cloning model according to claim 1, characterized in that: The prosody extraction module includes a pre-trained BERT model, a mask predictor, a word predictor and a multiplier, the mask predictor includes a fourth fully connected layer, the word predictor includes a fifth fully connected layer, the output features of the pre-trained BERT model are respectively input into the mask predictor and the word predictor, the output features of the mask predictor and the output features of the word predictor are input into the multiplier, and the output features of the prosody extraction module are obtained; The training process of the prosody extraction module includes: preprocessing the speech data in the public speech data set and extracting the corresponding MFCC features, and then inputting them into the prosody extraction module for training until convergence, so as to obtain a trained prosody extraction module.

5. The method for training a voice cloning model according to claim 1, characterized in that: The speech style diffusion module includes a first spectral normalization layer, a residual block sequence layer, a fifth Relu activation function layer, a second spectral normalization layer, an average pooling layer, a sixth Relu activation function layer and a sixth fully connected layer connected in sequence, the residual block sequence layer includes four residual modules connected in sequence, the residual module includes a third spectral normalization layer, a seventh Relu activation function layer, a fourth spectral normalization layer, a ninth convolutional layer, an eighth Relu activation function layer and a fifth spectral normalization layer connected in sequence, and the input features of the residual module are added to the output features of the fifth spectral normalization layer to obtain the output features of the residual module.

6. A training device for a speech cloning model, characterized in that: include: A data processing module is configured to collect a plurality of single-speaker speech data and a plurality of multi-speaker speech data and perform preprocessing respectively to obtain a first training data set and a second training data set, wherein the speech duration of the speech data in each piece of training data in the first training data set and the second training data set is a preset duration; A component construction module is configured to construct a text alignment module, a first pitch extraction module, a second pitch extraction module and a rhythm extraction module and train them respectively to obtain a trained text alignment module, a trained first pitch extraction module, a trained second pitch extraction module and a trained rhythm extraction module; a model construction module configured to construct a voice cloning model by combining the trained text alignment module, the trained first pitch extraction module, the trained second pitch extraction module and the trained prosody extraction module with a voice style diffusion module, a decoder and a random duration prediction module; train the voice cloning model using the first training data set to obtain a trained voice cloning model; and fine-tune the trained voice cloning model using the second training data set to obtain a fine-tuned voice cloning model; During the training and fine-tuning process of the speech cloning model, the first MFCC feature and the second MFCC feature are respectively extracted from the speech data and the reference speech in the first training data set or the second training data set, the first MFCC feature and the second MFCC feature are respectively input into the trained first pitch extraction module and the trained second pitch extraction module, the output feature of the trained first pitch extraction module and the output feature of the trained second pitch extraction module are input into the speech style diffusion module to obtain the first feature, the first MFCC feature and the phonemes corresponding to the text data in the first training data set or the second training data set are input into the trained text alignment module to obtain the second feature, the phonemes corresponding to the text data in the first training data set or the second training data set are sequentially passed through the trained prosody extraction module and the random duration prediction module to obtain the third feature, the first feature, the second feature and the third feature are input into the decoder to obtain the cloned speech, the cloned speech and the first MFCC feature are respectively input into the general speech pre-training model for adversarial training, and the parameters of the speech style diffusion module, the decoder and the random duration prediction module are updated by back propagation.

7. A voice cloning method, characterized in that: The fine-tuned voice cloning model obtained by training using the voice cloning model training method according to any one of claims 1 to 5 comprises the following steps: Acquire speech data of a target speaker and text data to be cloned, extract phonemes corresponding to the text data to be cloned, and the effective duration of the speech data is at least a preset duration; The speech data of the target speaker is used as a reference speech and the phonemes corresponding to the text data to be cloned are input into the fine-tuned speech cloning model. During the inference process of the fine-tuned speech cloning model, the corresponding MFCC features are extracted from the reference speech. The MFCC features corresponding to the reference speech are sequentially passed through the trained second pitch extraction module and the speech style diffusion module to obtain a fourth feature. The phonemes corresponding to the text data to be cloned are input into the trained text alignment module to obtain a fifth feature. The phonemes corresponding to the text data to be cloned are sequentially passed through the trained rhythm extraction module and the random duration prediction module to obtain a sixth feature. The fourth feature, the fifth feature and the sixth feature are input into the decoder to obtain the cloned speech.

8. A voice cloning device, characterized in that: The fine-tuned voice cloning model obtained by training using the voice cloning model training method according to any one of claims 1 to 5 comprises: A data acquisition module is configured to acquire speech data of a target speaker and text data to be cloned, and extract phonemes corresponding to the text data to be cloned, wherein the effective duration of the speech data is at least a preset duration; The cloning module is configured to input the speech data of the target speaker as the reference speech and the phonemes corresponding to the text data to be cloned into the fine-tuned speech cloning model, extract the corresponding MFCC features of the reference speech during the inference process of the fine-tuned speech cloning model, the MFCC features corresponding to the reference speech are sequentially passed through the trained second pitch extraction module and the speech style diffusion module to obtain a fourth feature, the phonemes corresponding to the text data to be cloned are input into the trained text alignment module to obtain a fifth feature, the phonemes corresponding to the text data to be cloned are sequentially passed through the trained rhythm extraction module and the random duration prediction module to obtain a sixth feature, and the fourth feature, the fifth feature and the sixth feature are input into the decoder to obtain the cloned speech.

9. An electronic device, comprising: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 5.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images