Voice cloning model generation method and device, storage medium and computer equipment

Abstract

The invention discloses a voice clone model generation method and device, a storage medium and computer equipment, relates to the technical field of voice synthesis and medical treatment, and mainly aims at solving the problems that a voice clone model trained by an existing method is relatively weak in noisy voice recognition capability and insufficient in robustness. The method mainly comprises the steps of obtaining a voice training sample set; a training model of the voice cloning model is constructed, and the training model of the voice cloning model comprises an initial speaker encoder, a self-supervised voice representation learning model and an initial voice synthesizer; training an initial speaker encoder and an initial voice synthesizer by using a natural voice training sample and a recognition result of a self-supervised voice representation learning model on the natural voice training sample; and extracting a speaker encoder and a voice synthesizer in the training model of the voice clone model, and generating the voice clone model according to the speaker encoder, the voice synthesizer and the text encoder. The method is mainly used for training a speech clone model.

Description

Technical Field

[0001] The present invention relates to the field of speech synthesis and medical technology, and in particular to a method and device for generating a speech cloning model, a storage medium, and a computer device. Background Art

[0002] With the rapid development of artificial intelligence and deep learning technology, speech synthesis technology has made significant progress. As an important branch of speech synthesis technology, the speech cloning model aims to generate highly similar speech output to a specific speaker by analyzing and imitating the voice characteristics of the speaker. As a cutting-edge technology in the field of artificial intelligence, speech synthesis technology has been applied in many fields in recent years. Especially in the medical field, speech cloning technology has shown broad application prospects in application scenarios such as personalized speech therapy, psychological counseling, and doctor-patient communication.

[0003] The existing technology mainly collects audio samples of the target speaker, extracts its voice features, and uses these features to train a neural network model. After training, the model can convert the input text into a synthetic speech similar to the target speaker's voice. However, for noisy speech data, the existing technology needs to first extract clean audio from the speech data, and then train the model based on labeled samples of noisy speech and clean speech. This model training method is not only difficult to construct training samples, but also the trained model has weak recognition ability for noisy speech and insufficient robustness. Summary of the invention

[0004] In view of this, the present invention provides a method and device for generating a speech cloning model, a storage medium, and a computer device, the main purpose of which is to solve the problem that the speech cloning model trained by the existing method has weak recognition ability for noisy speech and insufficient robustness.

[0005] According to one aspect of the present invention, a method for generating a voice cloning model is provided, comprising:

[0006] Acquire a speech training sample set, wherein the speech training sample set includes natural speech training samples that are not labeled with speech environment types;

[0007] Constructing a training model of a speech cloning model, wherein the training model of the speech cloning model includes an initial speaker encoder, a self-supervised speech representation learning model and an initial speech synthesizer;

[0008] Training the initial speaker encoder and the initial speech synthesizer using the natural speech training samples and the recognition results of the natural speech training samples by the self-supervised speech representation learning model;

[0009] When the training is completed, the speaker encoder and the speech synthesizer in the training model of the speech cloning model are extracted, and the speech cloning model is generated according to the speaker encoder, the speech synthesizer and the text encoder.

[0010] Furthermore, the using the natural speech training sample and the recognition result of the natural speech training sample by the self-supervised speech representation learning model to train the initial speaker encoder and the initial speech synthesizer includes:

[0011] Encoding the natural speech training sample by the initial speaker encoder to obtain an encoded output;

[0012] Predicting and processing the natural speech training sample by the self-supervised speech representation learning model to obtain a speech environment type and a speech unit representation, wherein the speech environment type includes a noisy environment and a quiet environment;

[0013] generating reconstructed speech by the initial speech synthesizer according to the coded output, the speech environment type and the speech unit representation;

[0014] Constructing a target loss function based on the encoded output and the reconstructed speech;

[0015] The initial speaker encoder and the initial speech synthesizer are trained using the target loss function to obtain a speaker encoder and a speech synthesizer.

[0016] Further, constructing a target loss function based on the encoded output and the reconstructed speech includes:

[0017] Constructing a cross entropy loss function and a contrast loss function respectively according to the encoding output;

[0018] Constructing a regularization term based on the reconstructed speech and the corresponding natural speech training sample;

[0019] The cross entropy loss function, the contrast loss function and the regularization term are weightedly summed to obtain a target loss function.

[0020] Furthermore, the encoding output includes a teacher network output and a student network output, and the cross entropy loss function and the contrast loss function are constructed according to the encoding output, respectively, including:

[0021] Subtract the exponential sliding average corresponding to the previous speaker's speech features from the teacher network output to obtain the processed teacher network output;

[0022] Constructing a cross entropy loss function based on the processed teacher network output and the student network output;

[0023] A contrast loss function is constructed based on the positive and negative speech feature samples corresponding to the encoded output and the natural speech training samples.

[0024] Furthermore, before generating the speech cloning model according to the speaker encoder, the speech synthesizer and the text encoder, the method further includes:

[0025] The clear speech data is processed by a self-supervised speech representation learning model to obtain a clear speech unit representation;

[0026] marking the clear speech data with the clear speech unit representation to obtain a text encoder training sample;

[0027] An initial speech cloning model is constructed based on the speaker encoder, the speech synthesizer and the initial text encoder;

[0028] The initial text encoder is independently trained according to the text encoder training sample to obtain a text encoder.

[0029] Furthermore, the self-supervised speech representation learning model is constructed based on the HuBERT model;

[0030] The speaker encoder is built based on the ECAPA-TDNN model;

[0031] The text encoder is built based on the XLM-RoBERTa model;

[0032] The speech synthesizer is an improved VITS model, which includes a post-processing encoder configured with speaker speech feature processing conditions, speech flow prediction conditions and speech duration prediction conditions.

[0033] Furthermore, after generating the speech cloning model according to the speaker encoder, the speech synthesizer and the text encoder, the method further includes:

[0034] Obtain reference speech data of the target speaker and target conversion text;

[0035] Encoding the reference speech data by the speaker encoder to obtain speech features of the target speaker;

[0036] Encoding the target conversion text by the text encoder to obtain text language features;

[0037] Based on the speech features of the target speaker and the language features of the text, a speech synthesizer is used to generate cloned speech data expressing the target conversion text in the speech of the target speaker.

[0038] According to another aspect of the present invention, a device for generating a voice cloning model is provided, comprising:

[0039] An acquisition module, configured to acquire a speech training sample set, wherein the speech training sample set includes natural speech training samples that are not labeled with speech environment types;

[0040] A construction module, used to construct a training model of a speech cloning model, wherein the training model of the speech cloning model includes an initial speaker encoder, a self-supervised speech representation learning model and an initial speech synthesizer;

[0041] A training module, configured to train the initial speaker encoder and the initial speech synthesizer by using the natural speech training samples and the recognition results of the natural speech training samples by the self-supervised speech representation learning model;

[0042] The generation module is used to extract the speaker encoder and speech synthesizer in the training model of the speech cloning model when the training is completed, and generate the speech cloning model according to the speaker encoder, the speech synthesizer and the text encoder.

[0043] Furthermore, the training module includes:

[0044] An encoding unit, configured to encode the natural speech training sample through the initial speaker encoder to obtain an encoded output;

[0045] A processing unit, configured to perform prediction processing on the natural speech training sample by using the self-supervised speech representation learning model to obtain a speech environment type and a speech unit representation, wherein the speech environment type includes a noisy environment and a quiet environment;

[0046] A generating unit, configured to generate reconstructed speech through the initial speech synthesizer according to the coded output, the speech environment type and the speech unit representation;

[0047] A construction unit, configured to construct a target loss function according to the encoded output and the reconstructed speech;

[0048] The initial speaker encoder and the initial speech synthesizer are trained using the target loss function to obtain a speaker encoder and a speech synthesizer.

[0049] Furthermore, in a specific application scenario, the construction unit is specifically used to construct a cross entropy loss function and a contrast loss function according to the encoding output; construct a regularization term according to the reconstructed speech and the corresponding natural speech training sample; and perform weighted summation of the cross entropy loss function, the contrast loss function and the regularization term to obtain a target loss function.

[0050] Further, in a specific application scenario, the construction unit is specifically used to subtract the exponential sliding average corresponding to the previous speaker's speech feature from the teacher network output to obtain a processed teacher network output;

[0051] Constructing a cross entropy loss function based on the processed teacher network output and the student network output;

[0052] A contrast loss function is constructed based on the positive and negative speech feature samples corresponding to the encoded output and the natural speech training samples.

[0053] Furthermore, the device also includes:

[0054] A processing module, for processing the clear speech data through a self-supervised speech representation learning model to obtain a clear speech unit representation;

[0055] A marking module, used for marking the clear speech data with the clear speech unit representation to obtain a text encoder training sample;

[0056] The construction module is further used to construct an initial speech cloning model based on the speaker encoder, the speech synthesizer and the initial text encoder;

[0057] The training module is further used to independently train the initial text encoder based on the text encoder training samples to obtain a text encoder.

[0058] Furthermore, in a specific application scenario, the self-supervised speech representation learning model is constructed based on the HuBERT model;

[0059] The speaker encoder is built based on the ECAPA-TDNN model;

[0060] The text encoder is built based on the XLM-RoBERTa model;

[0061] The speech synthesizer is an improved VITS model, which includes a post-processing encoder configured with speaker speech feature processing conditions, speech flow prediction conditions and speech duration prediction conditions.

[0062] Furthermore, the device also includes:

[0063] The acquisition module is also used for reference speech data of the target speaker and target conversion text;

[0064] A first encoding module, configured to encode the reference speech data through the speaker encoder to obtain speech features of the target speaker;

[0065] A second encoding module, used for encoding the target conversion text through the text encoder to obtain text language features;

[0066] The generating module is further used to generate cloned speech data expressing the target conversion text in the speech of the target speaker through a speech synthesizer according to the speech features of the target speaker and the language features of the text.

[0067] According to another aspect of the present invention, a storage medium is provided, wherein at least one executable instruction is stored in the storage medium, and the executable instruction enables a processor to execute operations corresponding to the above-mentioned method for generating a voice cloning model.

[0068] According to another aspect of the present invention, there is provided a computer device, comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus;

[0069] The memory is used to store at least one executable instruction, and the executable instruction enables the processor to execute operations corresponding to the above-mentioned method for generating a voice cloning model.

[0070] By means of the above technical solution, the technical solution provided by the embodiment of the present invention has at least the following advantages:

[0071] The present invention provides a method and device for generating a speech cloning model, a storage medium, and a computer device. First, a speech training sample set is obtained, wherein the speech training sample set includes natural speech training samples that are not labeled with speech environment types; a training model of a speech cloning model is constructed, wherein the training model of the speech cloning model includes an initial speaker encoder, a self-supervised speech representation learning model, and an initial speech synthesizer; the initial speaker encoder and the initial speech synthesizer are trained using the natural speech training sample and the recognition result of the natural speech training sample by the self-supervised speech representation learning model; when the training is completed, the speaker encoder and the speech synthesizer in the training model of the speech cloning model are extracted, and the speech cloning model is generated according to the speaker encoder, the speech synthesizer, and the text encoder. Compared with the prior art, the embodiment of the present invention distinguishes the environment type of natural speech training samples through a self-supervised speech representation learning model, and jointly trains the speaker encoder and speech synthesizer, which can improve the speech synthesizer's learning ability for noisy speech and avoid the situation where the initial beneficial characteristics of the speaker encoder (such as high speaker discrimination and robustness to noise) are forgotten, thereby improving the overall robustness of the speech cloning model and providing more accurate speech data support when the model is applied to various scenarios in the medical field. In addition, it can also reduce the difficulty of constructing speech training samples and improve the training efficiency of the model.

[0072] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0074] Figure 1 A flow chart of a method for generating a voice cloning model provided by an embodiment of the present invention is shown;

[0075] Figure 2 A flow chart of another method for generating a voice cloning model provided by an embodiment of the present invention is shown;

[0076] Figure 3 A block diagram of a device for generating a voice cloning model provided by an embodiment of the present invention is shown;

[0077] Figure 4 A schematic structural diagram of a computer device provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0078] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0079] The embodiment of the present invention provides a method for generating a voice cloning model, such as Figure 1 As shown, the method includes:

[0080] 101. Obtain a speech training sample set.

[0081] In an embodiment of the present invention, the speech training sample set includes natural speech training samples that are not labeled with speech environment types. The natural speech training samples are constructed based on original speech data obtained in real life and not subjected to denoising, and include both speech data in quiet environments and speech data in noisy environments. For example, the speech data of doctors in a noisy environment with many patients in a doctor's office and the speech data of doctors in a relatively quiet environment in a consulting room. Another example is the speech data of patients collected in a noisy environment in a ward and the speech data of patients collected in a quiet environment in a treatment room. In other words, in the training process of the speech cloning model, there is no need to denoise the speech data with the help of an external denoising model or an additional noise encoder to enhance the resistance to noise in the reference audio. Instead, the model can be trained directly based on speech data that has not been denoised and labeled, which greatly reduces the difficulty and cost of constructing the speech training sample set.

[0082] 102. Build a training model for the voice cloning model.

[0083] In an embodiment of the present invention, a training model of a speech cloning model having a different model structure from the speech cloning model in the inference stage is constructed to train the speaker encoder and speech synthesizer in the speech cloning model. The training model of the speech cloning model includes an initial speaker encoder, a self-supervised speech representation learning model and an initial speech synthesizer. Among them, the self-supervised speech representation learning model is a type of model that can learn effective speech representation from a large amount of unlabeled speech data. Self-supervised speech representation learning refers to the process of learning meaningful speech representation from speech only through a self-supervised learning method without using any label information. The self-supervised speech representation learning model has the ability to distinguish between noisy speech and clear speech without the need for specific noise training. The self-supervised speech representation learning model can be constructed based on models such as the Wav2Vec series model and the HuBERT (Hidden-Unit BERT) model.

[0084] 103. Train the initial speaker encoder and the initial speech synthesizer using the natural speech training samples and the recognition results of the natural speech training samples by the self-supervised speech representation learning model.

[0085] In an embodiment of the present invention, the speaker encoder and the speech synthesizer are jointly trained. The natural speech training sample is input into the self-supervised speech representation learning model so that the model learns the acoustic and language features of the natural speech training sample from the continuous speech data of the natural speech training sample, and distinguishes whether the currently input natural speech training sample belongs to noisy speech or clear speech, that is, the recognition result. The recognition result is input into the speech synthesizer together with the encoding result of the natural speech training sample as an environment type label of the natural speech training sample to obtain the loss function of the speech synthesizer. At the same time, the encoding output of the natural speech training sample on the speaker encoder side also corresponds to a loss function. The speaker encoder and the speech synthesizer are trained by the combination of two loss functions, which avoids the situation in which the speaker encoder is pre-trained separately in the existing method, and then the trained speaker encoder is placed in the speech cloning model for training, resulting in the initial beneficial characteristics of the speaker encoder (such as high speaker discrimination and robustness to noise) being forgotten.

[0086] In addition, based on the self-supervised speech representation learning model, the noisy speech in the natural speech training samples is recognized and processed. It does not require the speech training samples to have clear noise labels or noise perception mechanisms. It can understand the audio content through self-learning and distinguish between noisy speech and clear speech, ensuring that during training, the noisy speech input can be associated with the corresponding noisy speech output. That is, when there is noise in the input speech, during the speech synthesis process, the speech synthesizer will match the noisy reference speech with this input speech; when the input speech is very clear, during the speech synthesis process, the speech synthesizer will match the input speech with the clear reference speech. Thereby improving the overall robustness of the speech cloning model.

[0087] 104. When the training is completed, extract the speaker encoder and the speech synthesizer in the training model of the speech cloning model, and generate a speech cloning model based on the speaker encoder, the speech synthesizer and the text encoder.

[0088] In the embodiment of the present invention, after the training of the speaker encoder and the speech synthesizer is completed, the self-supervised speech representation learning model in the training model of the speech cloning model is replaced with a text editor to obtain a speech cloning model that can be used to generate cloned speech. The text editor can be a pre-trained model that has been independently trained. Since the speaker encoder and the speech synthesizer are obtained based on the self-supervised speech representation learning model under the training of natural speech training samples, they have good robustness for speech recognition in noisy environments. Therefore, the generated speech cloning model also has good learning ability and feature extraction ability. When facing reference speech data in a noisy environment in real life, the speech cloning model can maintain good feature extraction ability, thereby ensuring a high similarity between the synthesized cloned speech and the reference speech.

[0089] It should be noted that voice cloning technology can provide personalized voice output for patients with aphasia or speech disorders. By extracting the patient's voice features and generating new voices similar to their past voices, they can help them communicate with each other more naturally and confidently. The patient's voice extraction environment is sometimes not guaranteed. The voice cloning model trained by the above method can maintain good feature extraction capabilities and ensure a high degree of similarity between the synthesized patient clone voice and the patient's past voice, thereby ensuring the patient's user experience. During the rehabilitation process, speech therapists can use voice cloning to help patients practice voice and intonation. High-quality voice data can also help maximize the diagnosis and treatment effects of speech therapists.

[0090] In an embodiment of the present invention, for further explanation and limitation, as Figure 2 As shown, step 103 uses the natural speech training sample and the recognition result of the natural speech training sample by the self-supervised speech representation learning model to train the initial speaker encoder and the initial speech synthesizer, including:

[0091] 201. Encode the natural speech training sample by using the initial speaker encoder to obtain an encoded output.

[0092] 202. Predictively process the natural speech training sample using the self-supervised speech representation learning model to obtain a speech environment type and a speech unit representation.

[0093] 203. Generate reconstructed speech through the initial speech synthesizer according to the encoded output, the speech environment type and the speech unit representation.

[0094] 204. Construct a target loss function based on the encoded output and the reconstructed speech.

[0095] 205. Train the initial speaker encoder and the initial speech synthesizer using the target loss function to obtain a speaker encoder and a speech synthesizer.

[0096] In an embodiment of the present invention, a self-supervised loss function is constructed based on the encoding output, and a reconstruction loss function is constructed based on the reconstructed speech, and then a target loss function is constructed based on the self-supervised loss function and the reconstruction loss function. The encoding output is the speech features of the corresponding natural speech training samples obtained after the natural speech training samples are encoded by the initial speaker encoder. The reconstructed speech is based on the initial speech synthesizer with the encoding output as the parameter speech features, and the speech environment type and speech unit representation are used to reconstruct the speech data. The speech environment type includes a noisy environment and a quiet environment. The speech unit representation refers to a series of discrete, distinguishable units (such as phonemes, morphemes or higher-level speech units), which can be used for tasks such as speech synthesis and speech recognition.

[0097] In one embodiment of the present invention, for further explanation and limitation, the step of constructing a target loss function based on the encoded output and the reconstructed speech includes:

[0098] Constructing a cross entropy loss function and a contrast loss function respectively according to the encoding output;

[0099] Constructing a regularization term based on the reconstructed speech and the corresponding natural speech training sample;

[0100] The cross entropy loss function, the contrast loss function and the regularization term are weightedly summed to obtain a target loss function.

[0101] In the embodiment of the present invention, the loss function is constructed with the goal of maintaining the style adaptability of the speaker's features, that is, being able to capture the speaker's style and emotional changes, and improving the speaker's discrimination, that is, being able to distinguish the features of different speakers. A cross entropy loss function and a contrast loss function are constructed for the encoded output. The cross entropy loss function is used to ensure that the embeddings from the same speaker are close to each other, while the embeddings from different speakers are far away from each other. The contrast loss function is used to increase the diversity of embeddings to prevent the model from overconfidently gathering all embeddings to one point. A regularization term is constructed for the reconstructed speech to ensure that the distribution of the embedding has a certain smoothness and stability. The cross entropy loss function, the contrast loss function and the regularization term are weighted and summed by introducing a weight parameter that balances each loss to obtain a target loss function. Among them, corresponding weight parameters can be configured for the cross entropy loss function and the regularization term respectively, and the value of the weight parameter can be customized according to the training requirements of the voice cloning model and the specific application scenario requirements, and the embodiment of the present invention does not make specific limitations.

[0102] In one embodiment of the present invention, for further explanation and limitation, the step of constructing a cross entropy loss function and a contrast loss function according to the encoding output includes:

[0103] Subtract the exponential sliding average corresponding to the previous speaker's speech features from the teacher network output to obtain the processed teacher network output;

[0104] Constructing a cross entropy loss function based on the processed teacher network output and the student network output;

[0105] A contrast loss function is constructed based on the positive and negative speech feature samples corresponding to the encoded output and the natural speech training samples.

[0106] In an embodiment of the present invention, the encoding output includes the teacher network output and the student network output, that is, the speaker encoder includes the teacher network and the student network. Among them, in the framework of self-supervised learning, the teacher network (usually a pre-trained or more complex model) and the student network (the model being trained) play a key role. The teacher network provides a "target" or "supervision" signal, and the student network tries to imitate this signal. The cross entropy loss function is used to measure the difference between the output distributions of the teacher network and the student network on the same input (but with different enhancements). During the model training process, by minimizing the entropy loss between the output distributions of the teacher network and the student network on different enhanced random cuts of the same speech input, a flexible embedding space is provided for speaker recognition and sentiment analysis. This method not only improves the generalization ability of the model, but also enables the model to capture richer internal features of the speaker. In addition, before constructing the cross entropy loss function, the exponential sliding average corresponding to the previous speaker's speech features is subtracted from the teacher network output, and then the cross entropy loss function is constructed based on this result and the student network output. Among them, the previous speaker's speech features refer to the feature representation of a specific speaker that the model has learned before the current speech input is processed. Subtracting the previous speaker's speech features from the teacher network output helps the model capture changes in speaker features and update its internal feature representation. This operation can improve the ability to distinguish speaker features, allowing the model to more accurately identify different speakers and distinguish their speech features, thereby improving the ability to distinguish speaker features.

[0107] Positive speech feature samples are speech feature samples of the same speaker as the natural speech training samples. Natural negative speech feature samples are speech feature samples of different speakers corresponding to the natural speech training samples. The contrast loss function is used to characterize the loss between the speech features in the encoded output and the positive speech feature samples and the negative speech feature samples, respectively. The training goal is to minimize the loss with the positive speech feature samples and maximize the loss with the negative speech feature samples.

[0108] In an application example, the objective loss function construction process includes:

[0109] 1. Contrastive loss function construction: Given a pair of speech embeddings zi and zj, if they come from the same speaker, zi and zj should be close to each other; if they come from different speakers, they should be far away. This can be achieved by the following formula:

[0110]

[0111] Among them, sim(z i ,z j ) represents the speech embedding z i and z j is a similarity measure between (e.g., cosine similarity), T is a temperature parameter used to adjust the scale of the similarity scores. K is the number of feature embeddings in the batch, is the indicator function. The above formula ensures that when z i and z j When they come from the same speaker (i.e., they are positive pairs), their similarity scores will be maximized; and when z i and z j When they are from different speakers (i.e., they are negative pairs, their similarity scores are minimized.

[0112] 2. Construction of entropy loss function: According to the entropy loss function, the entropy functions of the student network output and the processed teacher network output can be calculated respectively, and then the cross entropy function of the student network output and the teacher network output can be determined. By constructing the entropy loss function, the uncertainty of speech embedding can be increased, encouraging the model to produce more dispersed speech embedding.

[0113]

[0114] Among them, p i is the probability distribution of speech embedding Z in the i-th cluster.

[0115] 3. Construction of regularization term:

[0116]

[0117] Here, λ is the regularization strength parameter.

[0118] 4. The target loss function is:

[0119] L=L contrastive +αL entropy +βL regularization (4);

[0120] Among them, α and β are weight parameters to balance various losses, L entropyRepresents the cross entropy loss function between the student network output and the teacher network output.

[0121] In one embodiment of the present invention, for further explanation and limitation, before the step of generating the speech cloning model according to the speaker encoder, the speech synthesizer and the text encoder, the method further includes:

[0122] The clear speech data is processed by a self-supervised speech representation learning model to obtain a clear speech unit representation;

[0123] marking the clear speech data with the clear speech unit representation to obtain a text encoder training sample;

[0124] An initial speech cloning model is constructed based on the speaker encoder, the speech synthesizer and the initial text encoder;

[0125] The initial text encoder is independently trained according to the text encoder training sample to obtain a text encoder.

[0126] In an embodiment of the present invention, the text encoder is independently trained on a training sample set of clear speech to predict the output of a self-supervised speech representation learning model. The clear speech data is high-quality, noise-free speech data. The text encoder mainly starts from the ARPABET phoneme dimension, takes the clear speech unit representation output by the self-supervised speech representation learning model as the target output, and takes the clear speech data as input to construct a text encoder training sample. During the training process of the initial text encoder, the model parameters of the speaker encoder and the speech synthesizer are not updated, and the initial text encoder is only independently trained under the overall model framework of the initial speech cloning model.

[0127] In one embodiment of the present invention, for further explanation and limitation, the self-supervised speech representation learning model is constructed based on the HuBERT model;

[0128] The speaker encoder is built based on the ECAPA-TDNN model;

[0129] The text encoder is built based on the XLM-RoBERTa model;

[0130] The speech synthesizer is an improved VITS model, which includes a post-processing encoder configured with speaker speech feature processing conditions, speech flow prediction conditions and speech duration prediction conditions.

[0131] In the embodiment of the present invention, the preferred model of the self-supervised speech representation learning model is the HuBERT model. The HuBERT model can understand the audio content by learning by itself, especially in distinguishing which are clear speech and which are noisy speech. In view of this feature of the model, the clone sound synthesis model is semi-supervised anti-noise training, and it is proved that this method is significantly better than the existing method based on automatic speech recognition (Automatic Speech Recognition, ASR) transcription in performance. The preferred model of the speaker encoder is the ECAPA-TDNN model. This model is a speaker recognition model based on deep learning, which has excellent performance in the field of voiceprint recognition. The preferred model of the text encoder is the XLM-RoBERTa model, which is a multilingual version of the RoBERTa model that can handle natural language understanding tasks in multiple languages. The speech synthesizer is a variant model of the VITS model, and the speaker speech feature processing conditions, speech stream prediction conditions and speech duration prediction conditions are added to its post-encoder to realize the analysis of the speaker speech features and the output features of the text encoder to generate cloned speech.

[0132] In one embodiment of the present invention, for further explanation and limitation, after generating the speech cloning model according to the speaker encoder, the speech synthesizer and the text encoder, the method further includes:

[0133] Obtain reference speech data of the target speaker and target conversion text;

[0134] Encoding the reference speech data by the speaker encoder to obtain speech features of the target speaker;

[0135] Encoding the target conversion text by the text encoder to obtain text language features;

[0136] Based on the speech features of the target speaker and the language features of the text, a speech synthesizer is used to generate cloned speech data expressing the target conversion text in the speech of the target speaker.

[0137] In the embodiment of the present invention, the reference voice data is the real voice data of the target speaker, and the target conversion text is the specific content that needs to be expressed using the voice of the cloned target speaker. Among them, the main function of the speaker encoder is to convert the voice of the target speaker into a vector representation, that is, the voice feature is also called voice embedding, which can represent the pitch, speaking speed, timbre, style and other characteristics of the speaker's voice. Through the voice representation generated by the encoder, the subsequent speech synthesizer can generate a voice similar to the original speaker. The text encoder maps the text content to the same representation space to obtain a symbolic representation for characterizing phonemes, that is, the text language feature. The speech synthesizer uses the text language features of the target conversion text as language input, and combines the voice features of the target speaker to synthesize voice data with a similar sound to the target speaker. In the application scenario of psychological counseling and treatment, more effective psychological counseling and treatment can be carried out by creating a human voice that is familiar to the patient. The high-similarity voice extracted by the voice cloning model can also make the patient have a stronger emotional resonance, thereby helping the patient to establish a stronger trust relationship and improve the treatment effect.

[0138] The present invention provides a method for generating a speech cloning model, firstly obtaining a speech training sample set, wherein the speech training sample set includes natural speech training samples that are not labeled with speech environment types; constructing a training model of a speech cloning model, wherein the training model of the speech cloning model includes an initial speaker encoder, a self-supervised speech representation learning model and an initial speech synthesizer; using the recognition results of the natural speech training samples and the self-supervised speech representation learning model on the natural speech training samples, the initial speaker encoder and the initial speech synthesizer are trained; when the training is completed, the speaker encoder and the speech synthesizer in the training model of the speech cloning model are extracted, and a speech cloning model is generated according to the speaker encoder, the speech synthesizer and the text encoder. Compared with the prior art, the embodiment of the present invention distinguishes the environment types of natural speech training samples through the self-supervised speech representation learning model, and jointly trains the speaker encoder and the speech synthesizer, so as to improve the learning ability of the speech synthesizer for noisy speech, and avoid the situation where the initial beneficial characteristics of the speaker encoder (such as high speaker discrimination and robustness to noise) are forgotten, thereby improving the overall robustness of the speech cloning model, and providing more accurate speech data support when the model is applied to various scenarios in the medical field. In addition, it can also reduce the difficulty of constructing speech training samples and improve the training efficiency of the model.

[0139] Furthermore, as a response to the above Figure 1 The embodiment of the present invention provides a device for generating a speech cloning model, such as Figure 3 As shown, the device comprises:

[0140] An acquisition module 31 is used to acquire a speech training sample set, wherein the speech training sample set includes natural speech training samples that are not labeled with speech environment types;

[0141] A construction module 32 is used to construct a training model of a speech cloning model, wherein the training model of the speech cloning model includes an initial speaker encoder, a self-supervised speech representation learning model and an initial speech synthesizer;

[0142] A training module 33, configured to train the initial speaker encoder and the initial speech synthesizer using the natural speech training samples and the recognition results of the natural speech training samples by the self-supervised speech representation learning model;

[0143] The generating module 34 is used to extract the speaker encoder and the speech synthesizer in the training model of the speech cloning model when the training is completed, and generate the speech cloning model according to the speaker encoder, the speech synthesizer and the text encoder.

[0144] Furthermore, the training module 33 includes:

[0145] An encoding unit, configured to encode the natural speech training sample through the initial speaker encoder to obtain an encoded output;

[0146] A processing unit, configured to perform prediction processing on the natural speech training sample by using the self-supervised speech representation learning model to obtain a speech environment type and a speech unit representation, wherein the speech environment type includes a noisy environment and a quiet environment;

[0147] A generating unit, configured to generate reconstructed speech through the initial speech synthesizer according to the coded output, the speech environment type and the speech unit representation;

[0148] A construction unit, configured to construct a target loss function according to the encoded output and the reconstructed speech;

[0149] The initial speaker encoder and the initial speech synthesizer are trained using the target loss function to obtain a speaker encoder and a speech synthesizer.

[0150] Furthermore, in a specific application scenario, the construction unit is specifically used to construct a cross entropy loss function and a contrast loss function according to the encoding output; construct a regularization term according to the reconstructed speech and the corresponding natural speech training sample; and perform weighted summation of the cross entropy loss function, the contrast loss function and the regularization term to obtain a target loss function.

[0151] Further, in a specific application scenario, the construction unit is specifically used to subtract the exponential sliding average corresponding to the previous speaker's speech feature from the teacher network output to obtain a processed teacher network output;

[0152] Constructing a cross entropy loss function based on the processed teacher network output and the student network output;

[0153] A contrast loss function is constructed based on the positive and negative speech feature samples corresponding to the encoded output and the natural speech training samples.

[0154] Furthermore, the device also includes:

[0155] A processing module, for processing the clear speech data through a self-supervised speech representation learning model to obtain a clear speech unit representation;

[0156] A marking module, used for marking the clear speech data with the clear speech unit representation to obtain a text encoder training sample;

[0157] The construction module 32 is further used to construct an initial speech cloning model according to the speaker encoder, the speech synthesizer and the initial text encoder;

[0158] The training module 33 is further used to independently train the initial text encoder according to the text encoder training samples to obtain a text encoder.

[0159] Furthermore, in a specific application scenario, the self-supervised speech representation learning model is constructed based on the HuBERT model;

[0160] The speaker encoder is built based on the ECAPA-TDNN model;

[0161] The text encoder is built based on the XLM-RoBERTa model;

[0162] The speech synthesizer is an improved VITS model, which includes a post-processing encoder configured with speaker speech feature processing conditions, speech flow prediction conditions and speech duration prediction conditions.

[0163] Furthermore, the device also includes:

[0164] The acquisition module 31 is also used for reference speech data of the target speaker and the target conversion text;

[0165] A first encoding module, configured to encode the reference speech data through the speaker encoder to obtain speech features of the target speaker;

[0166] A second encoding module, used for encoding the target conversion text through the text encoder to obtain text language features;

[0167] The generating module 34 is further configured to generate cloned speech data expressing the target converted text in the speech of the target speaker through a speech synthesizer according to the speech features of the target speaker and the language features of the text.

[0168] The invention provides a device for generating a speech cloning model. The device comprises the following steps: firstly obtaining a speech training sample set, wherein the speech training sample set comprises natural speech training samples which are not marked with speech environment types; constructing a training model of a speech cloning model, wherein the training model of the speech cloning model comprises an initial speaker encoder, a self-supervised speech representation learning model and an initial speech synthesizer; training the initial speaker encoder and the initial speech synthesizer using the recognition results of the natural speech training samples and the self-supervised speech representation learning model on the natural speech training samples; extracting the speaker encoder and the speech synthesizer in the training model of the speech cloning model after the training is completed, and generating a speech cloning model according to the speaker encoder, the speech synthesizer and the text encoder. Compared with the prior art, the embodiments of the present invention distinguish the environment types of natural speech training samples through a self-supervised speech representation learning model, and jointly train the speaker encoder and the speech synthesizer, which can improve the speech synthesizer's learning ability for noisy speech and avoid the initial beneficial characteristics of the speaker encoder (such as high speaker discrimination and robustness to noise) being forgotten, thereby improving the overall robustness of the speech cloning model. In addition, it can also reduce the difficulty of constructing speech training samples and improve the training efficiency of the model.

[0169] According to an embodiment of the present invention, a storage medium is provided, wherein the storage medium stores at least one executable instruction, and the computer executable instruction can execute the method for generating a voice cloning model in any of the above method embodiments.

[0170] Figure 4 A schematic diagram of the structure of a computer device provided according to an embodiment of the present invention is shown. The specific embodiment of the present invention does not limit the specific implementation of the computer device.

[0171] like Figure 4 As shown, the computer device may include: a processor (processor) 402 , a communication interface (Communications Interface) 404 , a memory (memory) 406 , and a communication bus 408 .

[0172] The processor 402 , the communication interface 404 , and the memory 406 communicate with each other via a communication bus 408 .

[0173] The communication interface 404 is used to communicate with other devices such as clients or other servers.

[0174] The processor 402 is used to execute the program 410, and specifically can execute the relevant steps in the above-mentioned embodiment of the method for generating a voice cloning model.

[0175] Specifically, the program 410 may include program codes, which include computer operation instructions.

[0176] The processor 402 may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention. The computer device includes one or more processors, which may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0177] The memory 406 is used to store the program 410. The memory 406 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0178] The program 410 may be specifically configured to enable the processor 402 to perform the following operations:

[0179] Acquire a speech training sample set, wherein the speech training sample set includes natural speech training samples that are not labeled with speech environment types;

[0180] Constructing a training model of a speech cloning model, wherein the training model of the speech cloning model includes an initial speaker encoder, a self-supervised speech representation learning model and an initial speech synthesizer;

[0181] Training the initial speaker encoder and the initial speech synthesizer using the natural speech training samples and the recognition results of the natural speech training samples by the self-supervised speech representation learning model;

[0182] When the training is completed, the speaker encoder and the speech synthesizer in the training model of the speech cloning model are extracted, and the speech cloning model is generated according to the speaker encoder, the speech synthesizer and the text encoder.

[0183] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0184] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for generating a voice cloning model, characterized in that: include: Acquire a speech training sample set, wherein the speech training sample set includes natural speech training samples that are not labeled with speech environment types; Constructing a training model of a speech cloning model, wherein the training model of the speech cloning model includes an initial speaker encoder, a self-supervised speech representation learning model and an initial speech synthesizer; Training the initial speaker encoder and the initial speech synthesizer using the natural speech training samples and the recognition results of the natural speech training samples by the self-supervised speech representation learning model; When the training is completed, the speaker encoder and the speech synthesizer in the training model of the speech cloning model are extracted, and the speech cloning model is generated according to the speaker encoder, the speech synthesizer and the text encoder.

2. The method according to claim 1, characterized in that The method of training the initial speaker encoder and the initial speech synthesizer by using the natural speech training sample and the recognition result of the natural speech training sample by the self-supervised speech representation learning model comprises: Encoding the natural speech training sample by the initial speaker encoder to obtain an encoded output; Predicting and processing the natural speech training sample by the self-supervised speech representation learning model to obtain a speech environment type and a speech unit representation, wherein the speech environment type includes a noisy environment and a quiet environment; generating reconstructed speech by the initial speech synthesizer according to the coded output, the speech environment type and the speech unit representation; Constructing a target loss function based on the encoded output and the reconstructed speech; The initial speaker encoder and the initial speech synthesizer are trained using the target loss function to obtain a speaker encoder and a speech synthesizer.

3. The method according to claim 2, characterized in that The constructing a target loss function according to the encoded output and the reconstructed speech comprises: Constructing a cross entropy loss function and a contrast loss function respectively according to the encoding output; Constructing a regularization term based on the reconstructed speech and the corresponding natural speech training sample; The cross entropy loss function, the contrast loss function and the regularization term are weightedly summed to obtain a target loss function.

4. The method according to claim 3, characterized in that The encoding output includes a teacher network output and a student network output, and the cross entropy loss function and the contrast loss function are constructed according to the encoding output, respectively, including: Subtract the exponential sliding average corresponding to the previous speaker's speech features from the teacher network output to obtain the processed teacher network output; Constructing a cross entropy loss function based on the processed teacher network output and the student network output; A contrast loss function is constructed based on the positive and negative speech feature samples corresponding to the encoded output and the natural speech training samples.

5. The method according to claim 1, characterized in that Before generating the speech cloning model according to the speaker encoder, the speech synthesizer and the text encoder, the method further includes: The clear speech data is processed by a self-supervised speech representation learning model to obtain a clear speech unit representation; marking the clear speech data with the clear speech unit representation to obtain a text encoder training sample; An initial speech cloning model is constructed based on the speaker encoder, the speech synthesizer and the initial text encoder; The initial text encoder is independently trained according to the text encoder training sample to obtain a text encoder.

6. The method according to claim 1, characterized in that The self-supervised speech representation learning model is built based on the HuBERT model; The speaker encoder is built based on the ECAPA-TDNN model; The text encoder is built based on the XLM-RoBERTa model; The speech synthesizer is an improved VITS model, which includes a post-processing encoder configured with speaker speech feature processing conditions, speech flow prediction conditions and speech duration prediction conditions.

7. The method according to any one of claims 1 to 6, characterized in that After generating the speech cloning model according to the speaker encoder, the speech synthesizer and the text encoder, the method further comprises: Obtain reference speech data of the target speaker and target conversion text; Encoding the reference speech data by the speaker encoder to obtain speech features of the target speaker; Encoding the target conversion text by the text encoder to obtain text language features; Based on the speech features of the target speaker and the language features of the text, a speech synthesizer is used to generate cloned speech data expressing the target converted text in the speech of the target speaker.

8. A device for generating a voice cloning model, characterized in that: include: An acquisition module, configured to acquire a speech training sample set, wherein the speech training sample set includes natural speech training samples that are not labeled with speech environment types; A construction module, used to construct a training model of a speech cloning model, wherein the training model of the speech cloning model includes an initial speaker encoder, a self-supervised speech representation learning model and an initial speech synthesizer; A training module, configured to train the initial speaker encoder and the initial speech synthesizer by using the natural speech training samples and the recognition results of the natural speech training samples by the self-supervised speech representation learning model; The generation module is used to extract the speaker encoder and speech synthesizer in the training model of the speech cloning model when the training is completed, and generate the speech cloning model according to the speaker encoder, the speech synthesizer and the text encoder.

9. A storage medium storing at least one executable instruction, wherein the executable instruction enables a processor to execute operations corresponding to the method for generating a voice cloning model according to any one of claims 1 to 7.

10. A computer device comprising: A processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to the method for generating a voice cloning model according to any one of claims 1 to 7.

Images