Voice cloning method and device, equipment and storage medium

Abstract

The invention relates to the technical field of artificial intelligence, and discloses a voice cloning method, which comprises the steps of extracting audio features of reference audio; and converting the audio text of the reference audio into an audio text phoneme sequence, and converting the reference text into a reference text phoneme sequence. And the style emotion prediction model processes the reference text phoneme sequence to obtain a style label or an emotion label. According to the audio features, the audio text phoneme sequence, the reference text phoneme sequence and the style label or emotion label of the reference text, clone audio features are synthesized, and the clone audio features are converted into clone audio. In the voice cloning process, the style emotion prediction model is adopted to predict and generate a style label or an emotion label closer to the reference text semantics, the label is used for guiding the process of synthesizing the clone audio characteristics, and the expressive force of the clone audio in the aspects of style and emotion can be improved. The invention also discloses a voice cloning device and equipment, and a storage medium.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, such as a method, apparatus, device, and storage medium for voice cloning. Background Technology

[0002] Voice cloning is a technique that extracts timbre from a speaker's voice and uses this timbre to synthesize target speech. Zero-sample voice cloning, as a method, extracts a fixed-dimensional vector from a small amount of reference audio provided by the speaker as a speaker representation, and then synthesizes cloned audio based on this representation. However, because the speaker representation only reflects the speaker's timbre and cannot capture their stylistic or emotional characteristics, the synthesized cloned audio cannot express style or emotion.

[0003] In related technologies, to enable cloned audio to express style or emotion, during speech cloning, in addition to extracting speaker representations from a small amount of reference audio provided by the speaker, style tags or emotion tags are also extracted from style and emotion reference audio. Transfer learning is then employed, using these style tags or emotion tags to guide the speech synthesis process based on speaker representations. This transfers the style or emotion from the style and emotion reference audio to the cloned audio, enabling the cloned audio to express style or emotion.

[0004] However, when performing voice cloning using related technologies, since the style or emotion of a style and emotion reference audio is transferred to the cloned audio, the expressiveness of the cloned audio synthesized using these technologies in terms of style or emotion is determined by the style or emotion expressed in the style and emotion reference audio. When the style and emotion reference audio lacks expression of style or emotion, the cloned audio is prone to poor expressiveness in style or emotion, thus reducing its overall expressiveness in terms of style or emotion. Summary of the Invention

[0005] To provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. This summary is not intended as a general commentary, nor is it intended to identify key / important components or describe the scope of protection of these embodiments, but rather as a prelude to the detailed description that follows.

[0006] This application provides a method, apparatus, device, and storage medium for voice cloning, which can enhance the expressiveness of cloned audio in terms of style or emotion.

[0007] In a first aspect, embodiments of this application provide a method for voice cloning, applied to an electronic device, the method comprising:

[0008] Feature extraction is performed on the reference audio input by the user to obtain the audio features of the reference audio;

[0009] The text of the reference audio is extracted using a speech recognition model to obtain the audio text of the reference audio.

[0010] The audio text of the reference audio is converted into an audio text phoneme sequence, and the reference text input by the user is converted into a reference text phoneme sequence;

[0011] The reference text phoneme sequence is processed using a trained style sentiment prediction model to obtain the style label or sentiment label of the reference text.

[0012] The audio features of the reference audio, the audio text phoneme sequence, the reference text phoneme sequence, and the reference text style tag or sentiment tag are input into the speech cloning model, so that the speech cloning model outputs cloned audio features;

[0013] The cloned audio features are converted into cloned audio.

[0014] Optionally, before processing the reference text phoneme sequence using the trained style sentiment prediction model, the method further includes: obtaining multiple audio training data from a preset dataset; inputting the multiple audio training data into a style sentiment classification model to extract features from the audio training data using the style sentiment classification model; initializing initial cluster centers for K categories; where K is the preset number of categories in the clustering process; clustering the features of the audio training data to obtain style labels or sentiment labels for the audio training data; and training the style sentiment prediction model using the style labels or sentiment labels of the audio training data and the phoneme sequences of each audio training data to obtain the trained style sentiment prediction model.

[0015] Optionally, initializing the initial cluster centers for the K categories includes: performing multiple sample selections based on the preset dataset to obtain multiple sample sets, each sample set including K groups of samples, and each group of samples including multiple audio data corresponding to the same speaker; inputting the multiple audio data into the style-emotion classification model, causing the style-emotion classification model to generate a style feature vector or emotion feature vector for each audio data; determining the sample similarity of each sample set based on the style feature vector or emotion feature vector of each audio data in each group of samples in each sample set; selecting the sample set with the lowest sample similarity from the multiple sample sets as the target sample set; and determining the average vector of each group of samples in the K groups of samples in the target sample set as the initial cluster centers for the K categories; wherein, the sample similarity of the sample set is used to characterize the degree of similarity between each group of samples in the sample set.

[0016] Optionally, determining the sample similarity of each sample set includes: averaging the style feature vector or emotion feature vector of each audio data in each group of samples in each sample set to obtain the average vector of each group of samples; calculating the cosine similarity between the average vectors of different groups of samples based on the average vector of each group of samples to obtain multiple cosine similarities; and calculating the sum of the multiple cosine similarities as the sample similarity of the sample set.

[0017] Optionally, feature extraction is performed on the user-input reference audio to obtain the audio features of the reference audio, including: inputting the reference audio into an audio codec model, so that the audio codec model outputs the audio features of the reference audio.

[0018] Optionally, converting the audio text of the reference audio into an audio text phoneme sequence includes: converting the audio text of the reference audio into an audio text phoneme sequence using a phoneme prediction model.

[0019] Optionally, converting the cloned audio features into cloned audio includes: inputting the cloned audio features into an audio codec model, causing the audio codec model to output the cloned audio.

[0020] Secondly, embodiments of this application provide a voice cloning device integrated into an electronic device, the device comprising:

[0021] The feature extraction module is used to extract features from the reference audio input by the user to obtain the audio features of the reference audio.

[0022] The text extraction module is used to extract the text of the reference audio using a speech recognition model, thereby obtaining the audio text of the reference audio.

[0023] A phoneme conversion module is used to convert the audio text of the reference audio into an audio text phoneme sequence, and to convert the reference text input by the user into a reference text phoneme sequence.

[0024] The label prediction module is used to process the phoneme sequence of the reference text using a trained style sentiment prediction model to obtain the style label or sentiment label of the reference text.

[0025] An audio cloning module is used to input the audio features of the reference audio, the audio text phoneme sequence, the reference text phoneme sequence, and the style tag or sentiment tag of the reference text into the speech cloning model, so that the speech cloning model outputs cloned audio features.

[0026] An audio conversion module is used to convert the cloned audio features into cloned audio.

[0027] Thirdly, embodiments of this application provide an electronic device, including a processor and a memory storing program instructions, wherein the processor is configured to execute the voice cloning method as described in the first aspect when running the program instructions.

[0028] Fourthly, embodiments of this application provide a storage medium storing program instructions, wherein the program instructions, when executed, perform the voice cloning method as described in the first aspect.

[0029] The voice cloning method, apparatus, device, and storage medium provided in this application can achieve the following technical effects:

[0030] In the speech cloning process, based on user-inputted reference audio and reference text, audio features of the reference audio are extracted, and the audio text of the reference audio is extracted using a speech recognition model. The audio text of the reference audio is then converted into an audio text phoneme sequence, and the reference text is also converted into a reference text phoneme sequence. For the reference text phoneme sequence, a pre-trained style and sentiment prediction model is used to process the reference text phoneme sequence to obtain style tags or sentiment tags for the reference text. The audio features of the reference audio, the audio text phoneme sequence, the reference text phoneme sequence, and the style tags or sentiment tags of the reference text are all input into the speech cloning model, causing the speech cloning model to output cloned audio features. These cloned audio features can be converted into cloned audio, thereby achieving speech cloning. In this embodiment, when obtaining style tags or sentiment tags during the speech cloning process, a pre-trained style and sentiment prediction model is used. That is, after processing the reference text phoneme sequence, the style and sentiment prediction model can predict and generate style tags or sentiment tags that are semantically closer to the reference text. Using this style tag or sentiment tag to guide the process of synthesizing cloned audio features in the speech cloning model can enable the cloned audio features output by the speech cloning model to accurately reflect the expressive intent of the reference text in terms of style or sentiment, thereby improving the expressiveness of the cloned audio obtained based on the cloned audio feature conversion in terms of style or sentiment.

[0031] The above general description and the description below are exemplary and illustrative only and are not intended to limit this application. Attached Figure Description

[0032] One or more embodiments are illustrated by way of example with reference to the accompanying drawings. These illustrations and drawings do not constitute a limitation on the embodiments. Elements having the same reference numerals in the drawings are shown as similar elements. The drawings are not to be scaled. And wherein:

[0033] Figure 1 This is a schematic diagram of an electronic device provided in an embodiment of this application;

[0034] Figure 2 This is a schematic diagram of a voice cloning method provided in an embodiment of this application;

[0035] Figure 3 This is a schematic diagram of a method for training a style sentiment prediction model provided in an embodiment of this application;

[0036] Figure 4 This is a schematic diagram of a voice cloning device provided in an embodiment of this application;

[0037] Figure 5 This is a schematic diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0038] The terms "first," "second," etc., used in the specification, claims, and drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this application described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion.

[0039] Unless otherwise stated, in the embodiments of this application, the term "multiple" means two or more.

[0040] In this embodiment, the character " / " indicates that the preceding and following objects are in an "or" relationship. For example, A / B means: A or B. The term "and / or" describes an association relationship between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or, A and B. The term "correspondence" can refer to an association or binding relationship; A corresponding to B means that there is an association or binding relationship between A and B.

[0041] To provide a more detailed understanding of the features and technical content of the embodiments of this application, the implementation of the embodiments of this application will be described in detail below with reference to the accompanying drawings. The accompanying drawings are for illustrative purposes only and are not intended to limit the embodiments of this application. In the following technical description, for ease of explanation, several details are used to provide a full understanding of the disclosed embodiments. However, one or more embodiments may still be implemented without these details. In other cases, well-known structures and devices may be simplified in their depiction to simplify the drawings.

[0042] As described in the background section, current speech cloning methods mainly include few-shot speech cloning and zero-shot speech cloning. Few-shot speech cloning involves fine-tuning a model using a small number of training samples to obtain a trained model. This trained model is then used to process the speaker's recorded audio data to achieve speech cloning. Zero-shot speech cloning, on the other hand, requires no fine-tuning. It directly extracts a fixed-dimensional vector from a small amount of reference audio provided by the speaker as a speaker representation, and synthesizes the speaker's cloned audio based on this representation. Since the speaker representation only reflects the speaker's timbre and cannot reflect their style or emotional characteristics, the synthesized cloned audio cannot express style or emotion.

[0043] To enable cloned audio to express style or emotion, the main techniques employed in speech cloning are as follows: Specifically, during speech cloning, in addition to extracting speaker representations from a small amount of reference audio provided by the speaker, style tags or emotion tags are extracted from style and emotion reference audio. Then, using transfer learning, these style tags or emotion tags guide the speech synthesis process based on the speaker representations, thereby transferring the style or emotion from the style and emotion reference audio to the cloned audio, enabling the cloned audio to express style or emotion.

[0044] However, when performing voice cloning using related technologies, since the style or emotion of a style and emotion reference audio is transferred to the cloned audio, the expressiveness of the cloned audio synthesized using these technologies in terms of style or emotion is determined by the style or emotion expressed in the style and emotion reference audio. When the style and emotion reference audio lacks style or emotion expression, the cloned audio is prone to poor style or emotion expression, thus reducing its overall expressiveness in terms of style or emotion.

[0045] Therefore, embodiments of this application provide a method, apparatus, device, and storage medium for speech cloning. During speech cloning, when using style tags or sentiment tags to guide the speech cloning model in speech synthesis, a pre-trained style and sentiment prediction model processes the phoneme sequence of the reference text to obtain style tags or sentiment tags that closely match the expressive intent of the reference text. Using these style tags or sentiment tags to guide the speech cloning model in synthesizing speech can improve the stylistic or emotional expressiveness of the synthesized cloned audio.

[0046] In this embodiment, the entity executing voice cloning can be an electronic device, such as a mobile phone, tablet, laptop, desktop computer, smart interactive flat panel, or other device with voice cloning functionality. The voice cloning function of the electronic device can be implemented using an application running on the device. When a user needs to use the electronic device for voice cloning, reference audio and reference text can be provided to the device, enabling it to clone the desired audio based on these. The reference audio can be the user's own speech, collected in real-time by the electronic device, or it can be speech not belonging to the user, pre-recorded or uploaded by the user. The reference text represents the content of the audio to be cloned. The user can customize the content of the reference text according to their needs. For example, if a user needs to clone an audio clip for advertising, the reference text can be advertising copy set by the user. Of course, the voice cloning method provided in this embodiment is not limited to advertising scenarios; it can also be used in other scenarios, such as news broadcasting, storytelling, or teacher lecturing. This embodiment does not limit the application scenario of the voice cloning method.

[0047] Combination Figure 1 As shown, the aforementioned electronic device 10 integrates multiple algorithm models. When performing speech cloning based on user-provided reference audio and text, the electronic device can utilize these integrated algorithm models. The multiple algorithm models integrated into the electronic device 10 include at least an audio encoding / decoding model 11, a speech recognition model 12, a phoneme prediction model 13, a style / sentiment prediction model 14, a speech cloning model 15, and a style / sentiment classification model 16. Specifically, the audio encoding / decoding model 11 is used to convert between audio files and feature sequences. The speech recognition model 12 is used to extract the text content of the audio. The phoneme prediction model 13 is used to map the text content into a phoneme sequence. The style / sentiment prediction model 14 is used to predict style tags or sentiment tags. The speech cloning model 15 is used to synthesize speech. The style / sentiment classification model 16 is used to convert audio data into feature vectors.

[0048] The following describes the method of voice cloning performed by electronic device 10 using various algorithm models.

[0049] Combination Figure 2 As shown in the figure, this application provides a method for voice cloning, which can be applied to the above-mentioned electronic device. The method includes the following steps:

[0050] S21, the electronic device extracts features from the reference audio input by the user to obtain the audio features of the reference audio.

[0051] In step S21, the audio features include at least the timbre, pitch, and loudness of the reference audio. Subsequently, based on the audio features of the reference audio, a clone audio with a high degree of similarity to the reference audio can be simulated and generated.

[0052] S22, the electronic device uses a speech recognition model to extract the text of the reference audio and obtain the audio text of the reference audio.

[0053] S23, the electronic device converts the audio text of the reference audio into an audio text phoneme sequence, and converts the reference text input by the user into a reference text phoneme sequence.

[0054] In steps S22 and S23, since the subsequent speech cloning model needs to synthesize speech based on the input phoneme sequence, phoneme conversion needs to be performed to facilitate speech synthesis by the speech cloning model. This converts the reference audio or reference text into a phoneme sequence. Because the phoneme sequence is formed based on text data conversion, the electronic device needs to convert the reference audio into audio text before converting the audio text of the reference audio into an audio text phoneme sequence. Specifically, the electronic device can use a speech recognition model to extract the text of the reference audio to obtain the audio text of the reference audio. The speech recognition model can be pre-trained; for example, it can be a model trained using the Automatic Speech Recognition (ASR) tool Kaldi, or it can be the open-source Whisper model.

[0055] S24, the electronic device uses a trained style sentiment prediction model to process the phoneme sequence of the reference text to obtain the style label or sentiment label of the reference text.

[0056] In step S24, the style sentiment prediction model adopts a Transformer architecture based on a self-attention mechanism. This architecture is an encoder-decoder network architecture, which includes multiple stacked Transformer Blocks, layer normalization, and linear layers. The structure of each Transformer Block includes at least multiple normalizations, multi-head attention mechanisms, and feedforward neural networks.

[0057] In step S24, the dimension of the reference text phoneme sequence input to the style sentiment prediction model is B*T*D, where B is the batch size, T is the number of frames of the audio features, and D is the feature dimension. The dimension of the style tag or sentiment tag output by the style sentiment prediction model is B*1. In the style sentiment prediction model, the dimension of the special marker CLS token (Classification Token) used for classification is B*1*D.

[0058] In step S24, during the application of the style sentiment prediction model, a CLS token can be appended before the time dimension of the reference text phoneme sequence, so that the CLS token and the reference text phoneme sequence serve as input together. During the training of the style sentiment prediction model, the hidden features corresponding to the CLS token are extracted for classification loss calculation.

[0059] In step S24, style tags are used to characterize the overall style of the reference text. Sentiment tags are used to characterize the sentiment of each segment of text within the reference text; a segment of text can be a sentence or a single word. For example, if the reference text is "How could you do this?", the sentiment tag "angry" can be predicted for the reference text. Similarly, if the reference text is "Today is July 12, 2024, the weather is sunny, and the high temperature is 32 degrees Celsius," the style tag "broadcast" can be predicted for the entire reference text.

[0060] S25, the electronic device inputs the audio features of the reference audio, the audio text phoneme sequence, the reference text phoneme sequence, and the style label or sentiment label of the reference text into the speech cloning model, so that the speech cloning model outputs cloned audio features.

[0061] In step S25, the network architecture of the speech clone model is similar to that of the style sentiment prediction model. The only difference is that a cross-attention mechanism is introduced in each Transformer Block of the encoder of the speech clone model. During the training of the speech clone model, the hidden features output by the multi-head attention mechanism are cross-attentioned with the style label or sentiment label to incorporate style sentiment information into the prediction process of the speech clone model without the need for an additional loss function.

[0062] S26, the electronic device converts the cloned audio features into cloned audio.

[0063] In step S26, converting cloned audio features into cloned audio means converting cloned audio features into audio waveforms.

[0064] The speech cloning method provided in this application involves extracting audio features from the reference audio based on user-inputted reference audio and reference text, extracting the audio text of the reference audio using a speech recognition model, converting the audio text of the reference audio into an audio text phoneme sequence, and converting the reference text into a reference text phoneme sequence. For the reference text phoneme sequence, a pre-trained style and sentiment prediction model is used to process the reference text phoneme sequence to obtain style tags or sentiment tags for the reference text. The audio features of the reference audio, the audio text phoneme sequence, the reference text phoneme sequence, and the style tags or sentiment tags of the reference text are all input into the speech cloning model, causing the speech cloning model to output cloned audio features. These cloned audio features can be converted into cloned audio, thereby achieving speech cloning. In this application embodiment, when obtaining style tags or sentiment tags during the speech cloning process, a pre-trained style and sentiment prediction model is used. That is, after processing the reference text phoneme sequence, the style and sentiment prediction model can predict and generate style tags or sentiment tags that are semantically closer to the reference text. Using this style tag or sentiment tag to guide the process of synthesizing cloned audio features in the speech cloning model can enable the cloned audio features output by the speech cloning model to accurately reflect the expressive intent of the reference text in terms of style or sentiment, thereby improving the expressiveness of the cloned audio obtained based on the cloned audio feature conversion in terms of style or sentiment.

[0065] Furthermore, the speech cloning method provided in this application, compared to related technologies where "the lack of style and emotion expression in the style and emotion reference audio easily leads to poor style and emotion expression in the cloned audio," employs a pre-trained style and emotion prediction model to directly obtain style tags or emotion tags based on the phoneme sequence of the reference text. This avoids obtaining style tags or emotion tags based on the reference audio, thus preventing the obtained style tags or emotion tags from being influenced by the style or emotion expression of the reference audio itself. Using the method described in this application, the obtained style tags or emotion tags can accurately reflect the expressive intent of the reference text in terms of style and emotion.

[0066] The process of steps S21 to S25 will be explained below.

[0067] In step S21, when the electronic device extracts features from the reference audio and obtains the audio features of the reference audio, it can input the reference audio into the audio codec model so that the audio codec model can extract and output the audio features of the reference audio.

[0068] In this embodiment, an audio codec model is used to extract audio features from the reference audio. These audio features are in token sequence format. Extracting audio features from the reference audio means converting the reference audio into a token sequence. The audio codec model can be a Neural Audio Codec, which employs an encoder-decoder architecture. Specifically, the Neural Audio Codec can be an EnCodec model, which includes a quantizer between the encoder and decoder. In this embodiment, when the Neural Audio Codec is an EnCodec model, the steps for extracting audio features from the reference audio are as follows: The reference audio is input to the encoder. The encoder encodes the reference audio to obtain a fixed-dimensional context vector. The quantizer uses Residual Vector Quantization (RVQ) to compress the context vector into a token sequence, thereby achieving feature extraction from the reference audio.

[0069] In this embodiment, when extracting audio features from reference audio, the audio codec model uses residual vector quantization, which can improve the efficiency of audio feature extraction.

[0070] In steps S22 and S23, after obtaining the audio text of the reference audio using the speech recognition model, the electronic device can further convert the audio text of the reference audio into an audio text phoneme sequence. Specifically, the electronic device converts the audio text of the reference audio into an audio text phoneme sequence using a phoneme prediction model.

[0071] For reference text, electronic devices can also directly convert the reference text into a reference text phoneme sequence. Specifically, electronic devices convert reference text into a reference text phoneme sequence through a phoneme prediction model.

[0072] In this embodiment, the principle by which the phoneme prediction model converts the audio text of a reference audio source into an audio text phoneme sequence is the same as the principle by which the phoneme prediction model converts the reference text into a reference text phoneme sequence. Specifically, the phoneme prediction model performs phoneme sequence conversion based on a pronunciation dictionary, which provides correct pronunciation information for words. By consulting the pronunciation dictionary, words in the text are decomposed into phonemes, thereby forming a phoneme sequence corresponding to the text.

[0073] In this embodiment, since speech synthesis is required based on phonemes during the speech cloning process, the text can be converted into phonemes in advance using a phoneme prediction model, which can provide the necessary input for the subsequent speech synthesis by the speech cloning model, thus ensuring the normal execution of speech synthesis.

[0074] In step S24, combined Figure 3 As shown, training a style sentiment prediction model for electronic devices includes the following steps:

[0075] S31: Obtain multiple audio training data from a preset dataset.

[0076] In step S31, the preset dataset includes a large amount of preset audio data. When training the style sentiment prediction model, multiple audio training data can be obtained from the preset dataset. The preset dataset includes not only audio data of various styles (e.g., news broadcasts, speeches, novel readings, poetry readings, etc.) but also audio data of various emotions (e.g., sadness, happiness, anger, etc.). Thus, given the rich style and emotion representation of the audio data in the preset dataset, training the style sentiment prediction model using the audio training data obtained from the preset dataset can improve the model's generalization ability.

[0077] S32, input multiple audio training data into the style sentiment classification model, so as to extract the features of each audio training data using the style sentiment classification model.

[0078] In step S32, the style and sentiment classification model can be the general language sentiment representation model emo2vec. In this embodiment, the hidden state of the last layer of the style and sentiment classification model is used as the feature vector before style and sentiment clustering, and the style and sentiment classification model generates a corresponding feature vector for each audio training data.

[0079] S33, initialize the initial cluster centers for K categories.

[0080] In step S33, K is the number of categories preset in the clustering process.

[0081] S34, cluster the features of each audio training data to obtain the style label or sentiment label of each audio training data.

[0082] In step S34, after clustering the features of each audio training data, the style label or sentiment label of the audio training data will be obtained.

[0083] In step S34, during the clustering of features of each audio training data, the cluster centers of each category will be updated. When the clustering of all audio training data is completed, the clustering process ends.

[0084] S35. The style and sentiment prediction model is trained using the style tags or sentiment tags of each audio training data and the phoneme sequence of each audio training data to obtain a trained style and sentiment prediction model.

[0085] In step S35, during the training of the style sentiment prediction model, the input is the style label or sentiment label of each audio training data, as well as the phoneme sequence of each audio training data. The output of the style sentiment prediction model is the style label or sentiment label of the phoneme sequence of each audio training data.

[0086] In step S35, after generating the phoneme sequence of each audio training data, the text of the audio training data can be extracted using a speech recognition model, and the text can be converted into a phoneme sequence using a phoneme prediction model.

[0087] In this embodiment, since the style and sentiment prediction model maps style and sentiment to the same feature space during application—that is, during the process of predicting and generating style or sentiment tags—it is also necessary to use the feature information of the audio training data in the style or sentiment dimension for unified iterative training when training the style and sentiment prediction model. In this way, the trained style and sentiment prediction model can predict and generate tags in the style or sentiment dimension based on the reference text.

[0088] Using this embodiment, during the training of the style sentiment prediction model, clustering can be employed to intelligently generate labels for each audio training data point, eliminating the need for manual labeling. This improves the efficiency and accuracy of labeling, thereby enhancing the efficiency and accuracy of training the style sentiment prediction model. Furthermore, since manual labeling is unnecessary, the complexity of training the style sentiment prediction model is also reduced.

[0089] Furthermore, the electronic device initializes the initial cluster centers for K categories, including: performing multiple sample selections based on a preset dataset to obtain multiple sample sets, each sample set comprising K groups of samples, and each group of samples including multiple audio data corresponding to the same speaker. The multiple audio data are input into a style-sentiment classification model, enabling the model to generate a style feature vector or sentiment feature vector for each audio data. Based on the style feature vector or sentiment feature vector of each audio data in each group of samples within each sample set, the sample similarity of each sample set is determined. From the multiple sample sets, the sample set with the lowest sample similarity is selected as the target sample set. The average vector of each group of samples in the target sample set is used to determine the initial cluster centers for the K categories. Here, the sample similarity of the sample set is used to characterize the degree of similarity between the groups of samples in the sample set.

[0090] In this embodiment, the style sentiment prediction model has K clusters. When determining the initial cluster centers for the K categories, a sample set of K groups of samples can be selected, with each group of samples corresponding to one category. Typically, the style sentiment prediction model uses the average vector of each group of samples in the initially selected sample set as the initial cluster center for each of the K categories. However, in this embodiment, to improve the clustering analysis capability of the style sentiment prediction model, multiple sample selections can be performed when determining the initial cluster centers for the K categories. Each time, a sample set including K groups of samples is selected, resulting in multiple sample sets. For each sample set, the similarity between the K groups of samples within the sample set is determined. Based on the sample similarity of each sample set, the target sample set with the lowest sample similarity is selected, and the initial cluster centers are determined using the target sample set with the lowest sample similarity.

[0091] By employing this embodiment, the initial cluster centers of each category in the style sentiment prediction model are determined by screening the target sample set with the lowest sample similarity. This avoids situations where the similarity between categories in the style sentiment prediction model is high, thereby improving the identification of different categories during the training process of the style sentiment prediction model, thus improving the accuracy of the cluster analysis of the style sentiment prediction model, and ultimately enhancing the cluster analysis capability of the style sentiment prediction model.

[0092] Furthermore, in this embodiment, the electronic device determines the sample similarity of each sample set based on the style feature vector or sentiment feature vector of each audio data in each group of samples. This includes: averaging the style feature vector or sentiment feature vector of each audio data in each group of samples to obtain the average vector of each group of samples; calculating the cosine similarity between the average vectors of different groups of samples based on the average vector of each group of samples to obtain multiple cosine similarities; and calculating the sum of the multiple cosine similarities as the sample similarity of the sample set.

[0093] The style and sentiment classification model generates a feature vector for each audio data point.

[0094] In this embodiment, the sample similarity of the sample set refers to the degree of similarity between different groups of samples in the sample set. After processing multiple audio data points in a certain group of samples using a style-sentiment classification model, the style-sentiment classification model generates a style feature vector and a sentiment feature vector for each audio data point. For multiple audio data points, multiple style feature vectors and multiple sentiment feature vectors are generated accordingly. The average vector of the group of samples is obtained by averaging the multiple style feature vectors and multiple sentiment feature vectors. Since the multiple audio data points in each group of samples correspond to the same speaker, the sentiment or style reflected by the multiple audio data points in each group of samples is relatively similar. Thus, the style and sentiment of all audio data points in the group of samples can be represented by the average vector of the group of samples. Furthermore, an average vector is calculated for each group of samples in the sample set, thereby obtaining K average vectors corresponding to K groups of samples.

[0095] The similarity between different groups of samples in a sample set is represented by the cosine similarity between the average vectors of the samples from different groups. A higher cosine similarity value indicates a greater degree of similarity between the different groups. For example, if the sample set includes group A and group B, with average vector 'a' for group A and average vector 'b' for group B, the similarity between group A and group B can be represented by the cosine similarity between vectors 'a' and 'b'.

[0096] For K groups of samples in the sample set, after calculating the average vector of each group of samples, for the K groups of samples in the sample set, calculate M = K! / (K-2)! * 2, where M is the number of combinations between different groups of samples in the K groups of samples, and a cosine similarity is calculated for each combination, thus obtaining M cosine similarities. The sum of the M cosine similarities represents the sample similarity of the entire sample set.

[0097] In this embodiment, since multiple audio data points in each of the K groups of samples correspond to the same speaker, the clustering categories are divided at the speaker level during the training of the style and sentiment prediction model. When determining the initial cluster centers, based on the characteristic that the styles and sentiments of the same speaker are not significantly different, the average style or average sentiment of the same speaker is used to calculate the sample similarity, and the average style or average sentiment of the same speaker is used as the cluster center. This simplifies the clustering process, improves clustering efficiency, and enhances the training efficiency of the style and sentiment prediction model.

[0098] Optionally, to improve the training efficiency of the style sentiment prediction model, when determining the initial cluster centers of the K categories in the style sentiment prediction pattern, only one sample selection is performed to obtain a sample set. In this way, the average vector of each of the K groups of samples in this sample set can be directly used as the initial cluster center for each of the K categories, without needing to calculate the sample similarity between multiple sample sets. This improves the efficiency of determining the initial cluster centers of the K categories, thereby improving the training efficiency of the style sentiment prediction model.

[0099] In steps S25 and S26, the electronic device inputs the audio features of the reference audio, the phoneme sequence of the audio text, the phoneme sequence of the reference text, and the style or sentiment tags of the reference text into the speech cloning model, causing the speech cloning model to output cloned audio features. During the output of cloned audio features, the style or sentiment tags guide the speech synthesis process. The cloned audio features can be converted into cloned audio, thereby achieving speech cloning.

[0100] In this embodiment, the speech synthesis process performed by the speech cloning model includes at least the following steps: First, based on the phoneme sequence (i.e., the audio text phoneme sequence and the reference text phoneme sequence), phoneme information, such as phoneme duration, frequency, and amplitude, is extracted from each phoneme. Then, according to the content of the reference text, the phoneme information is combined sequentially. During the combination process, the audio features of the reference audio and the style or sentiment tags of the reference text need to be considered, so that the synthesized speech better matches the expressive intent of the reference text.

[0101] Furthermore, the electronic device converts the cloned audio features into cloned audio, including: inputting the cloned audio features into an audio codec model, causing the audio codec model to output cloned audio.

[0102] In this embodiment, the audio codec model is a neural audio codec with the same network architecture as in step S21. The process of the neural audio codec model outputting cloned audio based on cloned audio features is the reverse of the process in step S21 where the neural audio codec model outputs the audio features of the reference audio based on the reference audio. In this embodiment, the neural audio codec model utilizes residual vector quantization and an encoder-decoder architecture to achieve the compression and reconstruction of cloned audio based on cloned audio features.

[0103] Of course, in this embodiment, the audio codec model can also be other network models with audio reconstruction capabilities, such as the neural vocoder HiFi-GAN (High Fidelity GAN, a high-fidelity speech synthesis model based on generative adversarial networks). In this embodiment, the audio codec model is not limited.

[0104] In this embodiment, during the process of the audio codec model outputting cloned audio, digital filtering and waveform superposition operations can also be performed to ensure the audio quality of the output cloned audio.

[0105] In this embodiment, since electronic devices cannot directly play cloned audio features, the cloned audio features can be converted into playable cloned audio signals so that users can play the cloned audio.

[0106] The following describes the device embodiments provided in this application.

[0107] Combination Figure 4 As shown, this application provides a speech cloning device, which is integrated into the electronic device in the aforementioned embodiments. The device includes a feature extraction module 401, a text extraction module 402, a phoneme conversion module 403, a tag prediction module 404, an audio cloning module 405, and an audio conversion module 406. Wherein:

[0108] The feature extraction module 401 is used to extract features from the reference audio input by the user to obtain the audio features of the reference audio.

[0109] The text extraction module 402 is used to extract the text of the reference audio using a speech recognition model, thereby obtaining the audio text of the reference audio.

[0110] The phoneme conversion module 403 is used to convert the audio text of a reference audio into an audio text phoneme sequence, and to convert the reference text input by the user into a reference text phoneme sequence.

[0111] The label prediction module 404 is used to process the phoneme sequence of the reference text using a trained style sentiment prediction model to obtain the style label or sentiment label of the reference text.

[0112] The audio cloning module 45 is used to input the audio features of the reference audio, the audio text phoneme sequence, the reference text phoneme sequence, and the style tag or sentiment tag of the reference text into the speech cloning model, so that the speech cloning model outputs cloned audio features.

[0113] Audio conversion module 406 is used to convert cloned audio features into cloned audio.

[0114] Optionally, in this embodiment, the speech cloning device further includes a model training module 407 for training a style-sentiment prediction model. The training process includes acquiring multiple audio training data from a preset dataset; inputting the multiple audio training data into a style-sentiment classification model to extract features from each audio training data; initializing initial cluster centers for K categories, where K is the preset number of categories in the clustering process; clustering the features of each audio training data to obtain style labels or sentiment labels for each audio training data; and training the style-sentiment prediction model using the style labels or sentiment labels of each audio training data, as well as the phoneme sequences of each audio training data, to obtain a trained style-sentiment prediction model.

[0115] Optionally, when initializing the initial cluster centers for the K categories in the model training module 407, it specifically performs multiple sample selections based on a preset dataset to obtain multiple sample sets. Each sample set includes K groups of samples, and each group of samples includes multiple audio data corresponding to the same speaker. These multiple audio data are input into the style-sentiment classification model, which generates a style feature vector or sentiment feature vector for each audio data. Based on the style feature vector or sentiment feature vector of each audio data in each group of samples within each sample set, the sample similarity of each sample set is determined. From the multiple sample sets, the sample set with the lowest sample similarity is selected as the target sample set. The average vector of each group of samples in the target sample set is used as the initial cluster centers for the K categories. Here, the sample similarity of the sample set is used to characterize the degree of similarity between the groups of samples in the sample set.

[0116] Optionally, when determining the sample similarity for each sample set, the model training module 407 is specifically used for...

[0117] The style feature vector or sentiment feature vector of each audio data point in each sample set is averaged to obtain the average vector of each sample set. Based on the average vector of each sample set, the cosine similarity between the average vectors of different sample sets is calculated, resulting in multiple cosine similarities. The sum of these multiple cosine similarities is taken as the sample similarity of the sample set.

[0118] Optionally, when the feature extraction module 401 extracts features from the reference audio to obtain the audio features of the reference audio, it is specifically used to input the reference audio into the audio codec model so that the audio codec model outputs the audio features of the reference audio.

[0119] Optionally, when converting the audio text of the reference audio into an audio text phoneme sequence, the phoneme conversion module 403 is specifically used to convert the audio text of the reference audio into an audio text phoneme sequence through a phoneme prediction model.

[0120] Optionally, when converting cloned audio features into cloned audio, the audio conversion module 406 is specifically used to input the cloned audio features into the audio codec model, so that the audio codec model outputs cloned audio.

[0121] In this embodiment, the voice cloning device performs the actions of the voice cloning method in the aforementioned embodiment, and its implementation principle or technical effect is similar, so it will not be described again here.

[0122] Combination Figure 5 As shown, this application embodiment provides an electronic device 500, including a processor 501 and a memory 502. Optionally, the device may further include a communication interface 503 and a bus 504. The processor 501, communication interface 503, and memory 502 can communicate with each other via the bus 504. The communication interface 503 can be used for information transmission. The processor 501 can call logical instructions in the memory 502 to execute the voice cloning method described in the above embodiment.

[0123] Furthermore, the logic instructions in the aforementioned memory 502 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium.

[0124] The memory 502, as a computer-readable storage medium, can be used to store software programs and computer-executable programs, such as program instructions / modules corresponding to the methods in the embodiments of this application. The processor 501 executes functional applications and data processing by running the program instructions / modules stored in the memory 502, that is, it implements the voice cloning method in the above embodiments.

[0125] The memory 502 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the terminal device. Furthermore, the memory 502 may include high-speed random access memory and may also include non-volatile memory.

[0126] This application provides a storage medium storing computer-executable instructions configured to execute the voice cloning method described in the above embodiments.

[0127] The aforementioned storage medium can be a transient computer-readable storage medium or a non-transitory computer-readable storage medium.

[0128] The technical solutions of this application embodiment can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes one or more instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in this application embodiment. The aforementioned storage medium can be a non-transitory storage medium, including: USB flash drive, portable hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, and other media capable of storing program code; it can also be a transient storage medium.

[0129] The foregoing description and accompanying drawings fully illustrate embodiments of this disclosure to enable those skilled in the art to practice them. Other embodiments may include structural, logical, electrical, procedural, and other changes. The embodiments represent only possible variations. Individual components and functions are optional unless explicitly required, and the order of operation may vary. Parts and features of some embodiments may be included in or replace parts and features of other embodiments. Moreover, the terminology used in this application is for describing embodiments only and is not intended to limit the claims. As used in the description of embodiments and claims, the singular forms “a,” “an,” and “the” are intended to equally include the plural forms unless the context clearly indicates otherwise. Similarly, the term “and / or” as used in this application means including one or more of the associated listed items and all possible combinations thereof. Additionally, when used in this application, the term "comprise" and its variations "comprises" and / or "comprising" refer to the presence of stated features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof. Without further limitations, an element defined by the phrase "comprises a..." does not exclude the presence of other identical elements in the process, method, or apparatus that includes said element. In this document, each embodiment may focus on the differences from other embodiments, and similar or identical parts between embodiments can be referred to mutually. For methods, products, etc., disclosed in the embodiments, if they correspond to the method section disclosed in the embodiments, the relevant parts can be referred to the description of the method section.

[0130] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0131] The methods and products (including but not limited to devices and equipment) disclosed in the embodiments herein can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of units may be merely a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to implement this embodiment according to actual needs. In addition, the functional units in the embodiments of this application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0132] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than that shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. In the descriptions corresponding to the flowcharts and block diagrams in the accompanying drawings, the operations or steps corresponding to different blocks may also occur in a different order than disclosed in the description; sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. Each block in a block diagram and / or flowchart, and combinations of blocks in a block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

Claims

1. A method for voice cloning, characterized in that, Applied to electronic devices, the method includes: Feature extraction is performed on the reference audio input by the user to obtain the audio features of the reference audio; The text of the reference audio is extracted using a speech recognition model to obtain the audio text of the reference audio. The audio text of the reference audio is converted into an audio text phoneme sequence, and the reference text input by the user is converted into a reference text phoneme sequence; The reference text phoneme sequence is processed using a trained style sentiment prediction model to obtain the style label or sentiment label of the reference text. The audio features of the reference audio, the phoneme sequence of the audio text, the phoneme sequence of the reference text, and the style tag or sentiment tag of the reference text are input into the speech cloning model, so that the speech cloning model outputs cloned audio features. The cloned audio features are converted into cloned audio.

2. The method according to claim 1, characterized in that, Before processing the reference text phoneme sequence using the trained style sentiment prediction model, the following steps are also included: Obtain multiple audio training data from a pre-set dataset; Multiple audio training data are input into the style sentiment classification model to extract features from the audio training data using the style sentiment classification model; Initialize K initial cluster centers for each category; where K is the preset number of categories in the clustering process; Clustering the features of the audio training data yields style tags or sentiment tags for the audio training data; The style and sentiment prediction model is trained using the style tags or sentiment tags of the audio training data and the phoneme sequence of the audio training data to obtain the trained style and sentiment prediction model.

3. The method according to claim 2, characterized in that, The initialization of the initial cluster centers for the K categories includes: Based on the preset dataset, multiple sample selections are performed to obtain multiple sample sets. Each sample set includes K groups of samples, and each group of samples includes multiple audio data corresponding to the same speaker. The multiple audio data are input into the style and sentiment classification model, so that the style and sentiment classification model generates a style feature vector or sentiment feature vector for each audio data. Based on the style feature vector or sentiment feature vector of each audio data in each group of samples in each sample set, the sample similarity of each sample set is determined respectively. From the multiple sample sets, the sample set with the lowest sample similarity is selected as the target sample set; The average vector of each of the K groups of samples in the target sample set is used as the initial cluster center for the K categories. The sample similarity of the sample set is used to characterize the degree of similarity between the samples in each group of the sample set.

4. The method according to claim 3, characterized in that, Determining the sample similarity for each of the sample sets includes: The average vector of each audio data point in each set of samples is calculated by averaging the style feature vector or emotion feature vector of each set of samples. Based on the average vector of each group of samples, calculate the cosine similarity between the average vectors of different groups of samples to obtain multiple cosine similarities. The sum of the multiple cosine similarities is calculated and used as the sample similarity of the sample set.

5. The method according to claim 1, characterized in that, The step of extracting features from the user-input reference audio to obtain the audio features of the reference audio includes: The reference audio is input into the audio codec model, which then outputs the audio features of the reference audio.

6. The method according to claim 1, characterized in that, Converting the audio text of the reference audio into an audio text phoneme sequence includes: The audio text of the reference audio is input into the phoneme prediction model, which then converts the audio text into an audio text phoneme sequence.

7. The method according to claim 1, characterized in that, Converting the cloned audio features into cloned audio includes: The cloned audio features are input into the audio codec model, which then outputs the cloned audio.

8. A device for voice cloning, characterized in that, Integrated into an electronic device, the device includes: The feature extraction module is used to extract features from the reference audio input by the user to obtain the audio features of the reference audio. The text extraction module is used to extract the text of the reference audio using a speech recognition model, thereby obtaining the audio text of the reference audio. A phoneme conversion module is used to convert the audio text of the reference audio into an audio text phoneme sequence, and to convert the reference text input by the user into a reference text phoneme sequence. The label prediction module is used to process the phoneme sequence of the reference text using a trained style sentiment prediction model to obtain the style label or sentiment label of the reference text. An audio cloning module is used to input the audio features of the reference audio, the audio text phoneme sequence, the reference text phoneme sequence, and the style tag or sentiment tag of the reference text into the speech cloning model, so that the speech cloning model outputs cloned audio features. An audio conversion module is used to convert the cloned audio features into cloned audio.

9. An electronic device comprising a processor and a memory storing program instructions, characterized in that, The processor is configured to perform the voice cloning method as described in any one of claims 1 to 7 when executing the program instructions.

10. A storage medium storing program instructions, characterized in that, When the program instructions are executed, they perform the voice cloning method as described in any one of claims 1 to 7.

Images