A speech cloning method and system based on cross-domain consistency loss

Abstract

The present application relates to a speech cloning method and system based on cross-domain consistency loss, belonging to the technical field of speech cloning, which solves the problem of low sound quality of existing speech cloning. It includes collecting the audio of the cloning object, obtaining the voiceprint features of the cloning object, inputting the audio into the pre-trained speech posterior graph model, and obtaining the speech posterior graph of the cloning object; based on the corpus and the speech posterior graph model, the source acoustic model is trained, the source acoustic model is transferred according to the speech posterior graph of the cloning object and the cross-domain consistency loss, and the adaptive acoustic model is obtained; based on the voiceprint features of the corpus and the corpus sample, the robust vocoder is trained; the phonemes and prosody of the text to be synthesized are obtained, input into the adaptive acoustic model to obtain the synthesized mel-frequency spectrum, the synthesized mel-frequency spectrum is input into the robust vocoder for speech synthesis, and the synthesized audio of the target cloning object is output according to the voiceprint features of the selected target cloning object. High-quality speech cloning is realized.

Description

Technical Field

[0001] This invention relates to the field of speech cloning technology, and in particular to a speech cloning method and system based on cross-domain consistency loss. Background Technology

[0002] Speech cloning refers to cloning the voice of a target object using a small amount of its audio. Typically, speech cloning technology can generate target audio with a pronunciation similar to the target object from any input text. The higher the similarity between the generated audio and the original audio, and the higher the naturalness and intelligibility of the generated audio, the better the speech cloning effect. With the rapid development of deep learning, speech synthesis systems based on deep neural networks can achieve effects comparable to human voices with the support of large amounts of high-quality corpora, drawing increasing attention to speech cloning technology.

[0003] Existing voice cloning systems all require paired <text, audio> data, which limits the application scenarios of voice cloning. When the text corresponding to the voice is missing, an ASR (Automatic Speech Recognition) system is typically used to first perform speech recognition on the audio to obtain the corresponding text. However, speech recognition systems may have inaccurate recognition of individual characters, which can significantly affect the effectiveness of voice cloning.

[0004] In existing technologies, the construction of acoustic models usually requires the use of a large amount of high-quality corpus. When a small amount of transfer learning is performed using cloned audio, the entire acoustic model is transferred, resulting in too many adaptive parameters, which is not conducive to commercial applications. Moreover, since there is a mismatch between the loss function used for model training and the final evaluation score, the decrease in the loss function cannot be used to determine whether the acoustic model has been trained successfully when using transfer learning. The model is prone to underfitting and overfitting.

[0005] Current vocoders also require training with a large corpus, which needs to include audio recordings of speakers of different ages, genders, vocal characteristics, and languages; the higher the diversity, the better. However, collecting a diverse corpus is time-consuming and labor-intensive in reality, and it still cannot perfectly fit speakers not present in the corpus. This results in relatively low sound quality synthesized by the speech cloning system. Summary of the Invention

[0006] Based on the above analysis, the embodiments of the present invention aim to provide a speech cloning method and system based on cross-domain consistency loss, in order to solve the problems of low sound quality and cumbersome speech cloning steps caused by the lack of text information and the inability of existing robust vocoders to fully fit the speaker features not appearing in the corpus.

[0007] On one hand, embodiments of the present invention provide a speech cloning method based on cross-domain consistency loss, comprising the following steps:

[0008] The audio of the cloned object is collected, the voiceprint features of the cloned object are obtained from the audio, and the audio of the cloned object is fed into the pre-trained speech posterior map model to obtain the speech posterior map of the cloned object.

[0009] Based on the corpus and speech posterior graph model, the acoustic model is trained to obtain the source acoustic model. The source acoustic model is then transferred and learned according to the speech posterior graph of the cloned object and cross-domain consistency loss to obtain the adaptive acoustic model.

[0010] Based on the voiceprint features of the corpus and corpus samples, a robust vocoder is obtained by training the vocoder.

[0011] The phonemes and prosody of the text to be synthesized are obtained, and the Mel spectrum to be synthesized is passed into the adaptive acoustic model. The Mel spectrum to be synthesized is then passed into the robust vocoder for speech synthesis. Based on the voiceprint features of the selected target clone object, the synthesized audio of the target clone object is output.

[0012] Based on further improvements to the above method, the acoustic model includes a phoneme and prosody encoder module, a speech posterior encoder module, a duration prediction module, a pitch prediction module, a volume prediction module, and a decoder module; wherein, the phoneme and prosody encoder module, the speech posterior encoder module, and the decoder module are all composed of multiple Transformer feedforward layers; the duration prediction module, the pitch prediction module, and the volume prediction module are all composed of multiple convolutional layers.

[0013] Based on the above method, a further improvement is made to the feedforward layer of each Transformer in the decoder module, which uses conditional layer normalization. The conditional layer normalization includes a speaker embedding layer and two linear layers, defined as follows:

[0014]

[0015] Where x is the vector before normalization, μ is the mean of x, σ is the variance of x, γ represents the learnable scalar in the conditional layer normalization, and β represents the learnable bias in the conditional layer normalization.

[0016] Based on further improvements to the above method, an acoustic model is trained using a corpus and a speech posterior graph model to obtain a source acoustic model, including:

[0017] Input the corpus samples into the pre-trained speech posterior graph model to obtain the speech posterior graph of the corpus samples;

[0018] The speech posterior map of the corpus sample is fed into the speech posterior map encoder module, and the phonemes and prosody of the corpus sample are fed into the phoneme and prosody encoder module.

[0019] The output of the phoneme and prosody encoder module and the speaker ID corresponding to the corpus sample are sequentially passed to the duration prediction module, pitch prediction module and volume prediction module to obtain the predicted values ​​of the pronunciation duration, pronunciation pitch and pronunciation volume of each phoneme in the corpus sample. The predicted values ​​and speaker ID are then passed to the decoder module to obtain the predicted Mel spectrum of the corpus sample.

[0020] Based on the errors between the outputs of the speech posterior graph encoder module and the phoneme and prosody encoder module, the errors between the predicted and actual values ​​of the duration, pitch, and volume of each phoneme in the corpus sample, and the errors between the predicted and actual Mel spectra, the source acoustic model is trained.

[0021] Based on further improvements to the above method, an adaptive acoustic model is obtained by transferring the source acoustic model from the cloned object's speech posterior map and cross-domain consistency loss, including:

[0022] Based on the trained source acoustic model, construct the target acoustic model; in the target acoustic model, except for the conditional layer normalization parameters in the feedforward layer of each Transformer in the decoder module, the other parameters are fixed.

[0023] The posterior image of the cloned object's speech is used as a learning sample and fed into the trained source acoustic model and target acoustic model. The distance between the feature spaces of the decoder modules in the source acoustic model and target acoustic model is calculated and used as the cross-domain consistency loss. This loss, together with the loss between the predicted Mel spectrum output by the target acoustic model and the real Mel spectrum of the learning sample, is used as the loss function to train the target acoustic model and obtain the adaptive acoustic model.

[0024] Based on the further improvement of the above method, the posterior image of the cloned object's speech is used as a learning sample and input into the source acoustic model and the target acoustic model. The distance between the feature spaces of the decoder modules in the source acoustic model and the target acoustic model is calculated, including:

[0025] In each layer of the decoder module, each learning sample is taken from the current batch in turn, and the cosine similarity between the current learning sample and other learning samples in the same batch is calculated in the feature space of the source acoustic model and the feature space of the target acoustic model, respectively. The probability distribution of the cosine similarity in the source acoustic model and the target acoustic model is calculated through the Softmax network layer, and the distance between the two probability distributions of the current learning sample in the current layer is obtained through KL divergence.

[0026] The average distance between the two probability distributions of all learning samples in the current batch at each layer is summed and used as the distance between the feature spaces of the decoder modules in the source acoustic model and the target acoustic model of the current batch.

[0027] Based on the above method, the vocoder is based on the HiFi-GAN model, which includes a generator module and a discriminator module. The generator module includes transposed convolutional layers with different receptive fields, and adds a one-dimensional convolution operation in each layer to map the voiceprint features into a voiceprint feature vector with the same dimension as the Mel spectrum latent variable features of the current layer.

[0028] Based on further improvements to the above method, a robust vocoder is obtained by training the vocoder using the voiceprint features of the corpus and corpus samples, including:

[0029] The Mel spectrum of the corpus samples is input into the vocoder. The corresponding voiceprint feature vector of the corpus sample mapped to the current layer is added to the Mel spectrum latent variable features of each layer of the generator module. The error of the generator module, the error of the discriminator module, the error of the Mel spectrum, and the feature matching error of each discriminator module are used as loss functions to train a robust vocoder.

[0030] Based on the further improvement of the above method, the Mel spectrum to be synthesized is fed into the robust vocoder for speech synthesis. According to the voiceprint features of the selected target clone object, the synthesized audio of the target clone object is output. This is achieved by adding the voiceprint feature vector of the selected target clone object mapped to the latent variable features of the Mel spectrum in each layer of the generator in the robust vocoder. Finally, the generator in the robust vocoder outputs the synthesized audio of the target clone object.

[0031] On the other hand, embodiments of the present invention provide a speech cloning system based on cross-domain consistency loss, comprising:

[0032] The audio processing module is used to collect the audio of the cloned object, extract the voiceprint features of the cloned object from the audio, and input the audio of the cloned object into the pre-trained speech posterior graph model to obtain the speech posterior graph of the cloned object.

[0033] The adaptive acoustic model training module is used to train an acoustic model based on a corpus and a speech posterior graph model to obtain a source acoustic model. The source acoustic model is then transferred and learned based on the speech posterior graph of the cloned object and cross-domain consistency loss to obtain an adaptive acoustic model.

[0034] The robust vocoder training module is used to train a vocoder based on the voiceprint features of the corpus and corpus samples to obtain a robust vocoder.

[0035] The speech and audio synthesis module is used to acquire the phonemes and prosody of the text to be synthesized, input the adaptive acoustic model to obtain the Mel spectrum to be synthesized, input the Mel spectrum to be synthesized into a robust vocoder for speech synthesis, and output the synthesized audio of the target clone object based on the voiceprint features of the selected target clone object.

[0036] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:

[0037] 1. To address the limitation of existing speech cloning systems requiring paired <text, audio> data, the cloning process first utilizes a pre-trained speech posterior graph model to obtain the speech posterior graph of the cloned object. Then, it uses the speech posterior graph and the speech posterior graph encoder module to transfer learn the conditional layer normalization of the decoder in the source acoustic model. This method has fewer adaptive parameters, faster training speed, and introduces cross-domain consistency loss, making the transfer learning more stable.

[0038] 2. By introducing voiceprint information to build an adaptive vocoder, high-quality results can still be obtained without transfer learning, which greatly improves efficiency and can fit the voices of speakers not found in the corpus, which is beneficial for commercial applications.

[0039] In this invention, the above-described technical solutions can be combined with each other to achieve more preferred combinations. Other features and advantages of this invention will be set forth in the following description, and some advantages may become apparent from the description or be learned by practicing the invention. The objects and other advantages of this invention can be realized and obtained from what is particularly pointed out in the description and drawings. Attached Figure Description

[0040] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts.

[0041] Figure 1 This is a flowchart of a speech cloning method based on cross-domain consistency loss in Embodiment 1 of the present invention;

[0042] Figure 2 This is an example diagram of text audio annotation in Embodiment 1 of the present invention;

[0043] Figure 3 This is a schematic diagram of the acoustic model structure in Embodiment 1 of the present invention;

[0044] Figure 4 This is a schematic diagram of the conditional layer normalization framework in the decoder in Embodiment 1 of the present invention;

[0045] Figure 5 This is a schematic diagram of the vocoder structure in Embodiment 1 of the present invention. Detailed Implementation

[0046] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.

[0047] Example 1

[0048] A specific embodiment of the present invention discloses a speech cloning method based on cross-domain consistency loss, such as... Figure 1 As shown, it includes the following steps:

[0049] S11: Collect the audio of the cloned object, extract the voiceprint features of the cloned object from the audio, and input the audio of the cloned object into the pre-trained speech posterior map model to obtain the speech posterior map of the cloned object.

[0050] It should be noted that this embodiment only requires the acquisition of a small amount of audio from the cloned object, ensuring that the audio is free of noise and reverberation. For example, 20 audio clips are obtained through public speeches, podcasts, or recordings by the cloned object. Preferably, the signal-to-noise ratio of each audio clip is not less than 35dB, and the duration of each audio clip is 5-15 seconds.

[0051] Furthermore, the ECAPA_TDNN model is used to obtain the voiceprint features of each audio file of the cloned object, and then the average value of the voiceprint features of each cloned object is calculated as the voiceprint features of each cloned object.

[0052] It should be noted that the speech posterior graph model adopts the speech posterior graph PPG (phonetic posterior grams) model. After pre-training the model with a large amount of audio from the corpus, the audio of the cloned object is input to obtain the speech posterior graph of the cloned object.

[0053] Compared with existing technologies, this embodiment only requires a small amount of audio from the cloned object, and there is no need to extract text from the audio or manually annotate the text. This avoids the problem of poor voice cloning effect caused by inaccurate text recognition and improves the efficiency of data processing.

[0054] S12: Based on the corpus and the speech posterior graph model, train the acoustic model to obtain the source acoustic model, and transfer learn the source acoustic model according to the speech posterior graph of the cloned object and the cross-domain consistency loss to obtain the adaptive acoustic model.

[0055] It should be noted that a corpus refers to a database containing a large amount of <text, audio> pairs of data, corresponding speaker IDs, and audio annotation information. The text consists of three parts: text number, text content, and corresponding pinyin, separated by "|". The text content includes three levels of prosodic markers, and the corresponding pinyin includes five levels of pinyin tone markers.

[0056] Specifically, the three levels of prosodic markers are #1, #2, and #3, where #1 represents the boundary of a prosodic word, #2 represents the boundary of a prosodic phrase, and #3 represents the boundary of a prosodic short sentence. The five levels of pinyin tone markers use Arabic numerals 1, 2, 3, 4, and 5 to represent the tones of pinyin. 1, 2, 3, and 4 represent the first, second, third, and fourth tones of pinyin, respectively, and 5 represents the neutral tone.

[0057] For example, a piece of text in the corpus may take the following form:

[0058] 000001|Hello #2 We #1 will use #1 SF Express #1 to mail #3 to you #1. |nin2hao3wo3men5jiang1yong4shun4feng1kuai4di4wei4nin2jin4xing2you2ji4

[0059] Record the speaker's audio in a quiet environment, based on the text. Preferably, ensure the audio is free of noise and reverberation, and that the signal-to-noise ratio of each recording is not less than 35dB. The audio should be recorded in uncompressed WAV format with a sampling rate of 41kHz.

[0060] Finally, the audio annotation tool Praat was used to annotate the text and audio, providing the corresponding text for each audio segment, as well as the pinyin-level and phoneme-level annotations for the audio. The pinyin-level annotations correspond to the three-level prosodic markers in the text, while the phoneme-level annotations map the three-level prosodic markers to the phoneme level using a mapping relationship between pinyin and phonemes. Based on the phoneme-level annotations, the phonemes and their prosodic levels can be obtained.

[0061] For example, Figure 2 This is a annotation for the text "Hello #2 We #1 will use #1 SF Express #1 to mail #3 to you #1" and the corresponding audio. Figure 2 In the text, the first line is the time domain of the audio, the second line is the frequency domain of the audio, the third line is the pinyin level label, and the fourth line is the phoneme level label.

[0062] It should be noted that the mapping relationship between pinyin and phonemes is pre-constructed. The construction method includes: first, constructing a phoneme set according to the pronunciation characteristics of speech, where each phoneme is the smallest unit, meaning that the pronunciation of each phoneme is different. Then, using the pronunciation characteristics of pinyin, the correspondence between pinyin and phonemes is constructed.

[0063] like Figure 3As shown, the acoustic model is used to convert linguistic information (phonemes and phoneme-level prosody) into acoustic information (Mel spectrum), including a phoneme and prosody encoder module, a speech posterior encoder module, a duration prediction module, a pitch prediction module, a volume prediction module, and a decoder module; among them, the phoneme and prosody encoder module, the speech posterior encoder module, and the decoder module are all composed of multiple Transformer feedforward layers; the duration prediction module, the pitch prediction module, and the volume prediction module are all composed of multiple convolutional layers.

[0064] The input to the phoneme and prosody encoder module is phonemes and prosody, and the input to the speech posterior map encoder module is a speech posterior map. The output of both modules is a latent vector with contextual information.

[0065] Preferably, the phoneme and prosody encoder module and the speech posterior map encoder module are composed of four Transformer feedforward layers, the decoder module is composed of six Transformer feedforward layers, and the duration prediction module, pitch prediction module and volume prediction module are each composed of two convolutional layers.

[0066] Furthermore, conditional normalization is used in the feedforward layer of each Transformer in the decoder module, such as... Figure 4 As shown, conditional layer normalization includes a speaker embedding layer and two linear layers, defined as follows:

[0067]

[0068] Where x is the vector before normalization, μ is the mean of x, σ is the variance of x, γ represents the learnable scalar in the conditional layer normalization, and β represents the learnable bias in the conditional layer normalization.

[0069] Based on the corpus and speech posterior graph model, the acoustic model is trained to obtain the source acoustic model, including:

[0070] ① Input the corpus samples into the pre-trained speech posterior graph model to obtain the speech posterior graph of the corpus samples.

[0071] ② Input the speech posterior map of the corpus sample into the speech posterior map encoder module, and input the phonemes and prosody of the corpus sample into the phoneme and prosody encoder module.

[0072] ③ The speaker ID corresponding to the corpus sample and the output of the phoneme and prosody encoder module are sequentially passed to the duration prediction module, pitch prediction module and volume prediction module to obtain the predicted values ​​of the pronunciation duration, pronunciation pitch and pronunciation volume of each phoneme in the corpus sample. The predicted values ​​and speaker ID are then passed to the decoder module to obtain the predicted Mel spectrum of the corpus sample.

[0073] It should be noted that when the output of the phoneme and prosody encoder module is not empty, the output of the phoneme and prosody encoder module and the speaker ID corresponding to the corpus sample are sequentially passed to the duration prediction module, pitch prediction module and volume prediction module. Otherwise, the output of the speech posterior map encoder module and the speaker ID corresponding to the corpus sample are sequentially passed to the duration prediction module, pitch prediction module and volume prediction module.

[0074] ④Based on the errors between the outputs of the speech posterior graph encoder module and the phoneme and prosody encoder module, the errors between the predicted and actual values ​​of the duration, pitch and volume of each phoneme in the corpus sample, and the errors between the predicted and actual Mel spectra, the source acoustic model is trained.

[0075] It should be noted that the error between the outputs of the speech posterior map encoder module and the phoneme and prosody encoder module is added to the loss function of the source acoustic model. This makes it easier to constrain the speech posterior map encoder and the phoneme and prosody encoder to a feature space, thus preparing for subsequent speech cloning.

[0076] Specifically, the optimization objective of the source acoustic model is expressed by the following formula:

[0077]

[0078] Where A is the source acoustic model during the training process, A s This represents the trained source acoustic model. The mean absolute error (MAE) between the predicted Mel spectrum and the true Mel spectrum is calculated. and These are the mean squared errors (MSE) between the predicted and actual values ​​for phoneme duration, pitch, and volume, respectively. λ represents the mean square error between the outputs of the speech posterior map encoder module and the phoneme and prosody encoder module. mel , λ d , λ p , λ e and λ ppg These are the coefficients for each error. For example, all error coefficients are set to 1.

[0079] It should be noted that in this embodiment, a large number of high-quality audio samples from the corpus are defined as the source domain, and a small number of audio samples from the cloned object are defined as the target domain. The transfer learning process transforms the feature space of the acoustic model in the source domain into the feature space of the target domain. However, during the transformation process, due to the extremely small amount of data in the target domain, and because there is a mismatch between the loss function used in model training and the final evaluation score—that is, a smaller loss function does not necessarily mean more fluent and natural speech—when using transfer learning to build an adaptive acoustic model, the decrease in the loss function cannot be used to determine whether the adaptive acoustic model has been trained successfully, as the model is prone to overfitting and underfitting. This embodiment introduces cross-domain consistency loss, which in transfer learning constrains the source acoustic model from overfitting the target domain data, thereby achieving the goal of stable training.

[0080] Specifically, the source acoustic model is transferred from the cloned object's speech posterior map and cross-domain consistency loss for learning. This involves continuing training the source acoustic model based on the cloned object's speech posterior map, and considering the consistency loss between the source and target domains during training to obtain an adaptive acoustic model, including:

[0081] ① Based on the trained source acoustic model, a target acoustic model is constructed. In the target acoustic model, except for the conditional layer normalization parameters in the feedforward layer of each Transformer in the decoder module, the other parameters are fixed. That is, the model parameters of the trained source acoustic model are used as the initial model parameters of the target acoustic model, and only the conditional layer normalization parameters in the decoder module are transferred and learned in the target acoustic model.

[0082] ② The posterior image of the cloned object's speech is used as a learning sample and fed into the trained source acoustic model and target acoustic model. The distance between the feature spaces of the decoder modules in the source acoustic model and target acoustic model is calculated and used as the cross-domain consistency loss. Together with the loss between the predicted Mel spectrum output by the target acoustic model and the real Mel spectrum of the learning sample, it is used as the loss function to train the target acoustic model and obtain the adaptive acoustic model.

[0083] It should be noted that during transfer learning, only the posterior image of the cloned object's speech is used. In the source acoustic model and the target acoustic model, learning samples are fed into the speech posterior image encoder module in batches. The speech posterior image encoder module outputs latent vectors with contextual information, which, together with the speaker ID, are input to the duration prediction module, pitch prediction module, and volume prediction module to obtain the predicted values ​​of the pronunciation duration, pronunciation pitch, and pronunciation volume of each phoneme, respectively. The predicted values ​​and the speaker ID are then fed into each layer of the decoder module, where the distance between the feature spaces of the decoder module in the source acoustic model and the target acoustic model is calculated.

[0084] Specifically, it includes:

[0085] In each layer of the decoder module, each learning sample is taken from the current batch in turn, and the cosine similarity between the current learning sample and other learning samples in the same batch is calculated in the feature space of the source acoustic model and the feature space of the target acoustic model, respectively. The probability distribution of the cosine similarity in the source acoustic model and the target acoustic model is calculated through the Softmax network layer, and the distance between the two probability distributions of the current learning sample in the current layer is obtained through KL divergence.

[0086] The average of the distances between the two probability distributions at each layer for all learning samples in the current batch is used as the distance between the feature spaces of the decoder modules in the source acoustic model and the target acoustic model of the current batch, i.e., the cross-domain consistency loss. The formula is shown below:

[0087]

[0088] Among them, A s→t Let A represent the target acoustic model. s D represents the trained source acoustic model. KL Let m represent the KL divergence distance, l represent the normalization of the l-th conditional layer in the acoustic model, and m represent the distance between the KL divergence and ... i P represents the learning sample i in the current batch. m (m) represents the feature space. represents the expected value, and in formula (3) it represents the average. This represents the probability distribution of the learning sample i in the l-th conditional layer of the source acoustic model, after normalization. The probability distribution of the learning sample i in the normalized conditional layer of the l-th layer of the target acoustic model is shown in the following formula:

[0089]

[0090] Here, sim(·) represents the cosine similarity function, and j represents the learning sample j in the same batch as i. The Softmax network layer is not part of the acoustic model; here, a separate Softmax network layer is used to calculate the probability distribution.

[0091] Adaptive acoustic model The optimization objective is expressed by the following formula:

[0092]

[0093] in, λ represents the mean absolute error between the predicted Mel spectrum output by the target acoustic model and the true Mel spectrum of the training samples. cd λ is the weighting coefficient for cross-domain consistency loss. melλ represents the weighting coefficient for the Mel-spectral loss. Preferably, λ cd For 200, λ mel It is 45.

[0094] After transfer learning is completed, the trained target acoustic model is used as an adaptive acoustic model in actual speech synthesis.

[0095] Compared with existing technologies, this embodiment utilizes a small amount of audio transfer learning from cloned objects to normalize the conditional layer of the decoder in the source acoustic model. It has fewer adaptive parameters, faster training speed, and introduces cross-domain consistency loss to avoid the source acoustic model from overfitting the target data, making the transfer learning more stable.

[0096] S13: Based on the voiceprint features of the corpus and corpus samples, train the vocoder to obtain a robust vocoder.

[0097] It should be noted that, as Figure 5 As shown, the vocoder is based on the HiFi-GAN model and includes a generator module and a discriminator module. The generator module includes transposed convolutional layers with different receptive fields, and adds a one-dimensional convolution operation to each layer. The discriminator module consists of a multi-period discriminator and a multi-scale discriminator. The transposed convolutional layers are used to upsample the Mel spectrum to obtain the audio waveform; the one-dimensional convolution operation is used to map the voiceprint features into a voiceprint feature vector with the same dimension as the Mel spectrum latent variable features of the current layer; the discriminator module is used to determine whether the audio is synthetic or real, thereby prompting the generator to generate more realistic audio.

[0098] For example, the Mel spectrum dimension of the input generator module is [T, 80], and the initial dimension of the voiceprint features is [1, 256]. The one-dimensional convolution in the first layer maps the voiceprint features to the [1, 80] dimension. Then, the voiceprint features are repeated T times to obtain voiceprint features of the [T, 80] dimension. Finally, the Mel spectrum latent variable features of the first layer are directly added to the mapped voiceprint features to obtain the output of the first layer, which is then fed into the second layer of the generator. The above steps are repeated to add the voiceprint features to each layer of the vocoder generator.

[0099] Based on the corpus, and according to the ECAPA_TDNN model, the speaker's voiceprint features corresponding to the corpus samples are obtained. Based on the corpus and the voiceprint features of the corpus samples, a robust vocoder is trained, including:

[0100] The Mel spectrum of the corpus samples is input into the vocoder. The corresponding voiceprint feature vector of the corpus sample mapped to the current layer is added to the Mel spectrum latent variable features of each layer of the generator module. The error of the generator module, the error of the discriminator module, the error of the Mel spectrum, and the feature matching error of each discriminator module are used as loss functions to train a robust vocoder.

[0101] Specifically, the loss function for the generator module is:

[0102]

[0103] Where G represents the generator module, D represents the discriminator module, and q represents the Mel spectrum.

[0104] The loss function of the discriminator module is:

[0105]

[0106] Where p represents the actual audio.

[0107] To ensure the stability of vocoder training, this embodiment adds a Mel-spectrum loss function and a feature matching loss function for the discriminator module, as shown below:

[0108]

[0109]

[0110] Where φ(·) is the function for calculating the Mel spectrum, T represents the number of discriminator modules, and D t and N t Let represent the features of the t-th discriminator module and the number of features, respectively.

[0111] Compared with existing technologies, this embodiment introduces the speaker's voiceprint features when training the vocoder, which can still obtain high-quality results without transfer learning, greatly improving efficiency.

[0112] S14: Obtain the phonemes and prosody of the text to be synthesized, input them into the adaptive acoustic model to obtain the Mel spectrum to be synthesized, input the Mel spectrum to be synthesized into the robust vocoder for speech synthesis, and output the synthesized audio of the target clone object according to the voiceprint features of the selected target clone object.

[0113] It should be noted that a prosody prediction model is constructed using a two-layer long short-term memory neural network (LSTM). After training, the prosody of the text to be synthesized can be obtained by inputting the text to be synthesized.

[0114] The text to be synthesized is converted into Pinyin, and the phonemes of the text to be synthesized are obtained according to the pre-built mapping relationship between Pinyin and phonemes.

[0115] The phonemes and prosody of the text to be synthesized are fed into the adaptive acoustic model to obtain the Mel spectrum of the selected target clone object. This Mel spectrum is then fed into the robust vocoder for speech synthesis. In the latent variable features of the Mel spectrum of the selected target clone object mapped to the current layer, the voiceprint feature vector of the generator in the robust vocoder is added to each layer of the generator. Finally, the generator in the robust vocoder outputs the synthesized audio of the target clone object.

[0116] Compared with existing technologies, this embodiment provides a speech cloning method based on cross-domain consistency loss. Addressing the limitation of existing speech cloning systems requiring paired <text, audio> data, this method first obtains the speech posterior map of the cloned object using a pre-trained speech posterior map model. Then, it utilizes the speech posterior map and the speech posterior map encoder module to transfer learn the conditional layer normalization of the decoder in the source acoustic model. This method requires fewer adaptive parameters, has a faster training speed, and introduces cross-domain consistency loss, making the transfer learning more stable. In actual speech synthesis, only the speaker's voiceprint features need to be changed to the voiceprint features of the cloned object to fit the voice of a speaker not found in the corpus, greatly improving speech synthesis efficiency and benefiting commercial applications.

[0117] Example 2

[0118] Another embodiment of the present invention discloses a speech cloning system based on cross-domain consistency loss, thereby implementing the speech cloning method based on cross-domain consistency loss in Embodiment 1. The specific implementation of each module is described in the corresponding description in Embodiment 1. The system includes:

[0119] The audio processing module is used to collect the audio of the cloned object, extract the voiceprint features of the cloned object from the audio, and input the audio of the cloned object into the pre-trained speech posterior graph model to obtain the speech posterior graph of the cloned object.

[0120] The adaptive acoustic model training module is used to train an acoustic model based on a corpus and a speech posterior graph model to obtain a source acoustic model. The source acoustic model is then transferred and learned based on the speech posterior graph of the cloned object and cross-domain consistency loss to obtain an adaptive acoustic model.

[0121] The robust vocoder training module is used to train a vocoder based on the voiceprint features of the corpus and corpus samples to obtain a robust vocoder.

[0122] The speech and audio synthesis module is used to acquire the phonemes and prosody of the text to be synthesized, input the adaptive acoustic model to obtain the Mel spectrum to be synthesized, input the Mel spectrum to be synthesized into a robust vocoder for speech synthesis, and output the synthesized audio of the target clone object based on the voiceprint features of the selected target clone object.

[0123] Preferably, the speech cloning system is built using an API service. By calling the API and inputting the text to be synthesized, it returns synthesized audio similar to the cloned object. Model training and the API interface are implemented using Python programming.

[0124] The quantitative evaluation metrics for the speech cloning system are Mean Opinion Score (MOS) and Similarity Mean Opinion Score (SMOS). MOS is used to evaluate the quality and fluency of the synthesized speech, with a score of 1-5. A higher score indicates a smoother and more natural synthesized speech. SMOS is mainly used to evaluate the similarity between the cloned speech and the original speech, with a score of 1-5. A higher score indicates a greater similarity between the cloned and original speech. The speech cloning system built based on this embodiment can achieve a MOS score of 4.17 and an SMOS score of 4.02.

[0125] Since the speech cloning system based on cross-domain consistency loss in this embodiment and the aforementioned speech cloning method based on cross-domain consistency loss can be mutually referenced, and this is a repetition of the previous description, it will not be repeated here. Because this system embodiment shares the same principle as the aforementioned method embodiment, it also possesses the corresponding technical effects of the aforementioned method embodiment.

[0126] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0127] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A speech cloning method based on cross-domain consistency loss, characterized in that, The method comprises the following steps: Collecting audio of a cloned object, obtaining a voiceprint feature of the cloned object from the audio, inputting the audio of the cloned object into a pre-trained speech posterior graph model, and obtaining a speech posterior graph of the cloned object; Training an acoustic model based on a corpus and the speech posterior graph model to obtain a source acoustic model; wherein the acoustic model comprises a phoneme and prosody encoder module, a speech posterior graph encoder module, a duration prediction module, a pitch prediction module, a volume prediction module, and a decoder module; the decoder module is composed of multiple feedforward layers of Transformers; corpus samples in the corpus are input into the pre-trained speech posterior graph model to obtain speech posterior graphs of the corpus samples, which are used to train the acoustic model; the source acoustic model is transferred and learned based on the speech posterior graph of the cloned object and a cross-domain consistency loss to obtain an adaptive acoustic model, comprising: constructing a target acoustic model based on the trained source acoustic model; in the target acoustic model, except for the conditional layer normalization parameters in the feedforward layers of each Transformer in the decoder module, other parameters are fixed; the speech posterior graph of the cloned object is input into the trained source acoustic model and the target acoustic model as a learning sample, the distance between the feature spaces of the decoder modules in the source acoustic model and the target acoustic model is calculated and used as a cross-domain consistency loss, which is used as a loss function together with the loss of the predicted mel spectrum output by the target acoustic model and the real mel spectrum of the learning sample to train the target acoustic model and obtain the adaptive acoustic model; Training a vocoder based on the voiceprint features of the corpus samples to obtain a robust vocoder; Obtaining phonemes and prosody of a text to be synthesized, inputting them into the adaptive acoustic model to obtain mel spectrum to be synthesized, inputting the mel spectrum to be synthesized into the robust vocoder for speech synthesis, and outputting synthesized audio of a target cloned object according to the voiceprint feature of the selected target cloned object.

2. The cross-domain consistency loss based speech cloning method of claim 1, wherein, In the acoustic model, the phoneme and prosody encoder module and the speech posterior graph encoder module are each composed of multiple feedforward layers of Transformers; the duration prediction module, the pitch prediction module, and the volume prediction module are each composed of multiple convolutional layers.

3. The cross-domain consistency loss based speech cloning method of claim 2, wherein, Conditional layer normalization is used in the feedforward layers of each Transformer in the decoder module, and the conditional layer normalization comprises a speaker embedding layer and two linear layers, which are defined as follows: ， in, The vector before normalization, for The mean, for variance This represents the learnable scalar in conditional layer normalization. This represents the learnable offset in conditional layer normalization.

4. The cross-domain consistency loss based speech cloning method of claim 3, wherein, The training of the acoustic model based on the corpus and the speech posterior graph model to obtain the source acoustic model comprises: Inputting the speech posterior graph of the corpus sample into the speech posterior graph encoder module and the phonemes and prosody of the corpus sample into the phoneme and prosody encoder module; Inputting the output of the phoneme and prosody encoder module and the speaker ID corresponding to the corpus sample into the duration prediction module, the pitch prediction module, and the volume prediction module in sequence to obtain predicted values of the duration, the pitch, and the volume of each phoneme of the corpus sample, respectively; and inputting the predicted values and the speaker ID into the decoder module to obtain the predicted mel spectrum of the corpus sample; The source acoustic model is trained according to errors between the voice posterior graph encoder module and the output of the phoneme and prosody encoder module, errors between predicted values and real values of phoneme duration, pronunciation pitch and pronunciation volume of the corpus sample, and errors between predicted mel spectrum and real mel spectrum.

5. The cross-domain consistency loss based speech cloning method of claim 1, wherein, The cloned object voice posterior graph is input into the source acoustic model and the target acoustic model as a learning sample, and a distance of a feature space of a decoder module in the source acoustic model and the target acoustic model is calculated, including: In each layer of the decoder module, each learning sample in the current batch is sequentially taken out, cosine similarity of the current learning sample and other learning samples in the same batch in the feature space of the source acoustic model and in the feature space of the target acoustic model is calculated, and probability distribution of the cosine similarity in the source acoustic model and the target acoustic model is calculated through a Softmax network layer, and a distance between the two probability distributions of the current learning sample in the current layer is obtained through KL divergence. Distances between the two probability distributions of all learning samples in the current batch in each layer are summarized and averaged as the distance of the feature space of the decoder module in the source acoustic model and the target acoustic model.

6. The cross-domain consistency loss based speech cloning method of claim 1, wherein, The vocoder is based on a HiFi-GAN model and includes a generator module and a discriminator module, wherein the generator module includes a transpose convolution layer with different receptive fields, and a one-dimensional convolution operation is added in each layer to map the voiceprint feature into a voiceprint feature vector with the same dimension as the mel spectrum hidden variable feature of the current layer.

7. The cross-domain consistency loss based speech cloning method of claim 1, wherein, The corpus and the voiceprint feature of the corpus sample are used to train the vocoder to obtain a robust vocoder, including: The mel spectrum of the corpus sample in the corpus is input into the vocoder, the voiceprint feature vector of the corresponding corpus sample in the current layer is added in the mel spectrum hidden variable feature of each layer of the generator module in sequence, the error of the generator module, the error of the discriminator module, the error of the mel spectrum, and the feature matching error of each discriminator module are used as a loss function, and the robust vocoder is trained.

8. The cross-domain consistency loss based speech cloning method of claim 1, wherein, The mel spectrum to be synthesized is input into the robust vocoder for voice synthesis, and the synthesized audio of the target cloned object is output according to the voiceprint feature of the selected target cloned object, which is the voiceprint feature vector of the selected target cloned object in the current layer mapped in the mel spectrum hidden variable feature of each layer of the generator in the robust vocoder, and finally the synthesized audio of the target cloned object is output by the generator in the robust vocoder.

9. A cross-domain consistency loss based speech cloning system configured to perform the cross-domain consistency loss based speech cloning method according to any one of claims 1-8. including: An audio processing module is configured to collect audio of a cloned object, obtain a voiceprint feature of the cloned object from the audio, input the audio of the cloned object into a pre-trained voice posterior graph model, and obtain a voice posterior graph of the cloned object; An adaptive acoustic model training module is configured to train an acoustic model to obtain a source acoustic model based on a corpus and the voice posterior graph model, and to obtain an adaptive acoustic model by transferring learning of the source acoustic model based on the voice posterior graph of the cloned object and a cross-domain consistency loss; A robust vocoder training module is configured to train a vocoder to obtain a robust vocoder based on a corpus and a voiceprint feature of a corpus sample. The voice audio synthesis module is configured to acquire phonemes and prosody of the text to be synthesized, input the phonemes and prosody into a self-adaptive acoustic model to obtain a mel-frequency spectrum to be synthesized, input the mel-frequency spectrum to be synthesized into a robust vocoder to perform voice synthesis, and output synthesized audio of the target cloned object according to the voiceprint features of the selected target cloned object.

Images