Voice cloning method

Abstract

The present application provides a voice cloning method, which relates to the field of artificial intelligence technology. The method comprises: obtaining a phoneme identifier corresponding to a text to be processed, a timbre identifier of a target object, and a mel spectrum corresponding to the voice of the target object; inputting the phoneme identifier, the timbre identifier, and the mel spectrum into an acoustic model, using a multi-head attention mechanism to extract the phoneme features of the text to be processed and the timbre features of the target object to calculate a first voice feature of each phoneme in the text to be processed; predicting the pronunciation duration of each phoneme according to the first voice feature of each phoneme and the phoneme feature of the text to be processed, to obtain a second voice feature of each phoneme; inputting the second voice feature of each phoneme into a decoding module of the acoustic model for decoding processing, to obtain a continuous voice feature of each phoneme; inputting the continuous voice feature of each phoneme into a vocoder model for voice synthesis, and outputting a cloned voice of the target object for the text to be processed.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a voice cloning method. Background Art

[0002] Voice cloning refers to synthesizing a voice that is very similar to the original speaker, which can be widely used in digital humans, virtual humans, video creation and other scenarios. In related technologies, text is usually converted into phonemes, which are converted into acoustic features, and then the voice of the target speaker is simulated based on the acoustic features and the Mel spectrum of the target speaker to obtain the cloned voice of the target speaker.

[0003] However, based on the above-mentioned voice cloning method, the cloned voice has a low similarity with the actual pronunciation of the target speaker. Summary of the invention

[0004] In order to solve the problem that the cloned voice has a low similarity with the actual pronunciation of the target speaker, the present application provides a voice cloning method, device, electronic device and storage medium.

[0005] In a first aspect, the present application provides a method for driving voice cloning, comprising:

[0006] Obtaining a phoneme identifier corresponding to the text to be processed, a timbre identifier of a target object, and a Mel spectrum corresponding to the speech of the target object;

[0007] Input the phoneme identifier, the timbre identifier, and the Mel spectrum into an acoustic model, and use an encoding module in the acoustic model to extract the phoneme features of the text to be processed and the timbre features of the target object;

[0008] The phoneme feature and the timbre feature are calculated using a multi-head attention mechanism to obtain a first speech feature of each phoneme in the text to be processed;

[0009] Predicting the pronunciation duration of each phoneme according to the first voice feature of each phoneme and the phoneme feature of the text to be processed, to obtain a second voice feature of each phoneme, wherein the second voice feature includes the pronunciation duration of each phoneme;

[0010] Inputting the second speech feature of each phoneme into the decoding module of the acoustic model for decoding processing to obtain the continuous speech feature of each phoneme;

[0011] The continuous speech feature of each phoneme is input into a vocoder model for speech synthesis, and a cloned speech of the target object for the text to be processed is output.

[0012] As an optional implementation of the embodiment of the present application, the encoding module of the acoustic model includes a phoneme encoder and a spectrum encoder; the phoneme identifier, the timbre identifier, and the mel spectrum are input into the acoustic model, and the encoding module of the acoustic model is used to extract the phoneme features of the text to be processed and the timbre features of the target object, including:

[0013] Inputting the phoneme identifier corresponding to the text to be processed and the timbre identifier of the target object into the phoneme encoder for feature extraction to obtain the phoneme features of the text to be processed;

[0014] The Mel spectrum corresponding to the speech of the target object is input into the spectrum encoder for feature extraction to obtain the timbre feature of the target object.

[0015] As an optional implementation of the embodiment of the present application, the acoustic model includes: an attention module; the multi-head attention mechanism is used to calculate the phoneme feature and the timbre feature to obtain the first speech feature of each phoneme in the text to be processed, including:

[0016] Input the phoneme features of the text to be processed as query content and the timbre features of the target object as keywords and content values ​​into the attention module;

[0017] The phoneme features of the text to be processed and the timbre features of the target object are calculated using a multi-head attention mechanism to obtain the first speech feature of each phoneme in the text to be processed.

[0018] As an optional implementation of the embodiment of the present application, the acoustic model includes: a duration predictor; the pronunciation duration of each phoneme is predicted according to the first speech feature of each phoneme and the phoneme feature of the text to be processed to obtain the second speech feature of each phoneme, including:

[0019] Extracting global timbre features of the Mel spectrum;

[0020] The feature obtained by adding the first speech feature of each phoneme in the text to be processed and the global timbre feature, as well as the phoneme feature of the text to be processed, are input into the duration predictor to predict the pronunciation duration of each phoneme, and the pronunciation duration of each phoneme is normalized to obtain and output the second speech feature of each phoneme.

[0021] As an optional implementation of the embodiment of the present application, the vocoder model includes: a multi-head vector quantizer and a HifiGAN vocoder; the continuous speech feature of each phoneme is input into the vocoder model for speech synthesis, and the cloned speech of the target object for the text to be processed is output, including:

[0022] Inputting the continuous speech feature of each phoneme into the multi-head vector quantizer for quantization processing to obtain the discrete speech feature of each phoneme;

[0023] The discrete speech features of each phoneme are input into the HifiGAN vocoder for speech synthesis, and the cloned speech of the target object for the text to be processed is obtained and output.

[0024] As an optional implementation of the embodiment of the present application, the obtaining of the phoneme identifier corresponding to the text to be processed includes:

[0025] Acquire a text to be processed, and convert characters in the text to be processed into text phonemes;

[0026] Based on a phoneme identifier dictionary, the text phonemes are converted into phoneme identifiers corresponding to the text to be processed.

[0027] As an optional implementation of the embodiment of the present application, the obtaining of the text to be processed and converting characters in the text to be processed into text phonemes includes:

[0028] Acquire a text to be processed, and if the text to be processed includes Chinese characters and English characters, separate the Chinese characters from the English characters, and add a rhythm level to the Chinese characters;

[0029] Converting the Chinese characters and the prosody levels into Chinese phonemes, wherein the Chinese phonemes include character phonemes corresponding to the Chinese characters and prosody phonemes corresponding to the prosody levels, and converting the English characters into English phonemes;

[0030] The Chinese phonemes and the English phonemes are concatenated to form text phonemes corresponding to the text to be processed.

[0031] In a second aspect, the present application provides a voice cloning device, comprising:

[0032] An acquisition module, used to acquire the phoneme identifier corresponding to the text to be processed, the timbre identifier of the target object, and the Mel spectrum corresponding to the speech of the target object;

[0033] An encoding module, used for inputting the phoneme identifier, the timbre identifier, and the Mel spectrum into an acoustic model, and extracting the phoneme features of the text to be processed and the timbre features of the target object by using the encoding module in the acoustic model;

[0034] A calculation module, used for calculating the phoneme feature and the timbre feature by using a multi-head attention mechanism to obtain a first speech feature of each phoneme in the text to be processed;

[0035] A processing module, configured to predict the pronunciation duration of each phoneme according to the first voice feature of each phoneme and the phoneme feature of the text to be processed, so as to obtain a second voice feature of each phoneme, wherein the second voice feature includes the pronunciation duration of each phoneme;

[0036] A decoding module, used for inputting the second speech feature of each phoneme into the decoding module of the acoustic model for decoding processing to obtain the continuous speech feature of each phoneme;

[0037] The synthesis module is used to input the continuous speech features of each phoneme into a vocoder model for speech synthesis, and output the cloned speech of the target object for the text to be processed.

[0038] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a memory and a processor, wherein the memory is used to store a computer program, and the processor is used to execute the voice cloning method described in the first aspect or any optional implementation manner of the first aspect when calling the computer program.

[0039] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the voice cloning method described in the first aspect or any optional implementation manner of the first aspect is implemented.

[0040] Compared with the prior art, the technical solution provided by the embodiments of the present application has the following advantages:

[0041] The speech cloning method provided by the embodiment of the present application includes: obtaining a phoneme identifier corresponding to a text to be processed, a timbre identifier of a target object, and a Mel spectrum corresponding to the speech of the target object; inputting the phoneme identifier, the timbre identifier, and the Mel spectrum into an acoustic model, and extracting the phoneme features of the text to be processed and the timbre features of the target object using an encoding module in the acoustic model; calculating the phoneme features and the timbre features using a multi-head attention mechanism to obtain a first speech feature of each phoneme in the text to be processed; predicting the pronunciation duration of each phoneme according to the first speech feature of each phoneme and the phoneme feature of the text to be processed to obtain a second speech feature of each phoneme, wherein the second speech feature includes the pronunciation duration of each phoneme; inputting the second speech feature of each phoneme into a decoding module of the acoustic model for decoding processing to obtain a continuous speech feature of each phoneme; inputting the continuous speech feature of each phoneme into a vocoder model for speech synthesis, and outputting a cloned speech of the target object for the text to be processed. In the embodiment of the present application, a multi-head attention mechanism is used to calculate the phoneme features of the text to be processed and the timbre features of the target object to obtain a first speech feature, and a second speech feature is obtained by performing duration prediction based on the first speech feature and the phoneme features of the text to be processed, that is, the second speech feature includes the phoneme features, timbre features, and duration features of the phonemes, which is equivalent to extracting the speech features of the target object for each phoneme from multiple angles, so that the speech synthesized by the vocoder model based on the continuous speech features of each phoneme is closer to the pronunciation of the target object, thereby improving the authenticity of the cloned speech. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0043] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0044] Figure 1 A schematic diagram of a process for generating a cloned voice corresponding to a text to be processed provided in one embodiment of the present application;

[0045] Figure 2 A flowchart of the steps of a voice cloning method provided in one embodiment of the present application;

[0046] Figure 3 A flowchart of a voice cloning method provided by one embodiment of the present application;

[0047] Figure 4 A structural block diagram of a voice cloning device provided in one embodiment of the present application;

[0048] Figure 5 An internal structure diagram of an electronic device provided for one embodiment of the present application. DETAILED DESCRIPTION

[0049] In order to make the purpose, implementation mode and advantages of the present application clearer, the exemplary implementation mode of the present application will be clearly and completely described below in conjunction with the drawings in the exemplary embodiments of the present application. Obviously, the described exemplary embodiments are only part of the embodiments of the present application, rather than all the embodiments.

[0050] Based on the exemplary embodiments described in this application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the claims attached to this application. In addition, although the disclosure in this application is introduced according to one or several exemplary examples, it should be understood that each aspect of these disclosures can also constitute a complete implementation method separately. It should be noted that the brief description of the terms in this application is only for the convenience of understanding the implementation methods described below, and is not intended to limit the implementation methods of this application. Unless otherwise specified, these terms should be understood according to their common and usual meanings.

[0051] The voice cloning method provided in the embodiment of the present application can be obtained based on a trained voice cloning model, where the voice cloning model includes an acoustic model and a vocoder model.

[0052] Reference Figure 1 As shown, Figure 1 A flow chart of generating cloned speech corresponding to a text to be processed provided in an embodiment of the present application. Exemplarily, after obtaining the text to be processed, the text to be processed is normalized, and the text normalization includes normalizing the punctuation in the text; when the text to be processed is a mixed text of Chinese and English, the Chinese and English characters are separated to obtain Chinese characters and English characters, and the English characters are converted into corresponding phonemes, and the rhythmic levels of the Chinese characters are proposed and then converted into corresponding phonemes; the phonemes corresponding to the Chinese characters and the phonemes corresponding to the English characters are mapped to corresponding phoneme identifiers; the phoneme identifier, the timbre identifier of the target object, and the voice of the target object are input into a trained voice cloning model for voice cloning, and the voice of the target object for the text to be processed is obtained and output.

[0053] In one embodiment, Figure 2 As shown, a voice cloning method is provided. Figure 2The flowchart of the steps of the voice cloning method provided by an embodiment of the present application includes the following steps S21-S26.

[0054] S21. Obtain the phoneme identifier corresponding to the text to be processed, the timbre identifier of the target object, and the Mel spectrogram corresponding to the voice of the target object.

[0055] The phoneme identifier is used to identify the corresponding text phoneme. The text to be processed in the embodiment of the present application can be a full Chinese text, a full English text, or a mixed text of Chinese and English.

[0056] Exemplarily, after obtaining the text to be processed, convert the characters in the text to be processed into text phonemes; based on the phoneme identifier dictionary, convert the text phonemes into the phoneme identifiers corresponding to the text to be processed. The text phonemes in this embodiment refer to all the phonemes corresponding to the characters obtained by converting the text. A phoneme is the smallest unit that makes up a syllable. For example, the character "中" in the text is converted into two phonemes "zh" and "ong1" after conversion.

[0057] Among them, when the text to be processed is a full Chinese text, normalize the punctuation marks in the text to be processed, convert the text such as numbers and dates in the text to be processed, and remove the punctuation and special characters other than commas, periods, semicolons, etc. Replace all other punctuation marks with the target punctuation mark, add a prosody level to the Chinese characters, convert the Chinese characters after adding the prosody level into corresponding phonemes, and then map the phonemes to the corresponding phoneme identifiers.

[0058] When the text to be processed is a full English text, normalize the punctuation marks in the text to be processed, convert the text such as numbers and dates in the text to be processed, and remove the punctuation and special characters other than commas, periods, semicolons, etc. Replace all other punctuation marks with the target punctuation mark, convert the English characters into corresponding phonemes, and then map the phonemes to the corresponding phoneme identifiers.

[0059] If the text to be processed includes Chinese characters and English characters, converting the characters in the text to be processed into text phonemes may include: separating the Chinese characters and the English characters, and adding a prosody level to the Chinese characters. The prosody level is used to represent the pronunciation duration of the Chinese characters; converting the Chinese characters and the prosody level into Chinese phonemes. The Chinese phonemes include the character phonemes corresponding to the Chinese characters and the prosody phonemes corresponding to the prosody level, converting the English characters into English phonemes; splicing the Chinese phonemes and the English phonemes to form the text phonemes corresponding to the text to be processed.

[0060] Among them, it also includes normalizing punctuation marks in the Chinese-English mixed text, converting texts such as numbers and dates in the text to be processed, removing punctuation and special characters other than commas, periods, semicolons, etc., and replacing other punctuation marks with target punctuation marks. The prosody levels can include four levels, such as: no-boundary level, prosodic word boundary level, prosodic phrase boundary level, and prosodic intonation boundary level. Prosody is the pronunciation rhythm and pattern of the corresponding sound of the text. For the same sentence, different prosodic structures correspond to different pronunciations, and each prosody level represents a different pause duration. That is, different prosody levels have differences in aspects such as stress and pause time. One prosody level also corresponds to one phoneme, that is, the text phoneme includes both the phoneme corresponding to the character and the phoneme corresponding to the prosody level. Exemplarily, after adding a prosody level to "中", three phonemes can be obtained through conversion.

[0061] Exemplarily, the prosody level corresponding to Chinese characters can be obtained based on the Chinese prosody model, and the Chinese prosody model is a classification model based on BERT. After obtaining the text phoneme, based on the phoneme identification dictionary, the text phoneme is converted into the phoneme identification corresponding to the text to be processed, and the mapping relationship between phonemes and phoneme identifications is recorded in the phoneme identification dictionary.

[0062] The target object refers to the real voice object corresponding to the cloned voice. For example, if the text to be processed is to be cloned into the voice of user A for playback, then user A is the target object. Obtaining the timbre identification of the target object can be obtained based on one or more voices of user A, which will not be elaborated in this embodiment of the present application.

[0063] The Mel spectrogram corresponding to the voice of the target object can be obtained based on one or more voices of the target object, which will not be elaborated in this embodiment of the present application.

[0064] S22. Input the phoneme identification, the timbre identification, and the Mel spectrogram into the acoustic model, and use the encoding module in the acoustic model to extract the phoneme features of the text to be processed and the timbre features of the target object.

[0065] Exemplarily, as shown in Figure 3 shown, Figure 3 is a schematic flowchart of a voice cloning method provided by an embodiment of the present application. The encoding module of the acoustic model includes a phoneme encoder and a spectrogram encoder.

[0066] Input the phoneme identification corresponding to the text to be processed and the timbre identification of the target object into the phoneme encoder for feature extraction to obtain the phoneme features of the text to be processed; input the Mel spectrogram corresponding to the voice of the target object into the spectrogram encoder for feature extraction to obtain the timbre features of the target object.

[0067] S23. Calculate the phoneme features and the timbre features using a multi-head attention mechanism to obtain a first speech feature of each phoneme in the text to be processed.

[0068] Exemplarily, the acoustic model includes: an attention module; the process of obtaining the first speech feature may include: inputting the phoneme features of the text to be processed as query content, and the timbre features of the target object as keywords and content values ​​into the attention module; using a multi-head attention mechanism to calculate the phoneme features of the text to be processed and the timbre features of the target object, to obtain the first speech feature of each phoneme in the text to be processed.

[0069] Exemplarily, the phoneme identifier corresponding to the text to be processed and the timbre identifier of the target object are input into the phoneme encoder for encoding, and the latent vector Hm is Q (Query, query content), and the timbre feature Gm of the target object obtained based on the Mel spectrum is K / V (Keys / Values, value) for multi-head attention calculation to obtain the first speech feature of each phoneme in the text to be processed, and the first speech feature is a speech feature at the phoneme level. This embodiment of the application does not repeat the multi-head attention mechanism in a similar manner.

[0070] S24. Predicting the pronunciation duration of each phoneme according to the first phoneme feature of each phoneme and the phoneme feature of the text to be processed, to obtain the second phoneme feature of each phoneme.

[0071] The second speech feature includes the pronunciation duration of each phoneme, and the pronunciation duration feature is included in the second speech feature.

[0072] Reference Figure 3 As shown, the acoustic model includes: a duration predictor; the pronunciation duration of each phoneme is predicted according to the first speech feature of each phoneme and the phoneme feature of the text to be processed to obtain the second speech feature of each phoneme, including:

[0073] Extract the global timbre feature of the Mel spectrum; input the feature obtained by adding the first speech feature of each phoneme in the text to be processed and the global timbre feature, and the phoneme feature of the text to be processed into the duration predictor, predict the pronunciation duration of each phoneme, and normalize the pronunciation duration of each phoneme to obtain and output the second speech feature of each phoneme.

[0074] Exemplarily, a global timbre feature is extracted based on the Mel spectrum corresponding to the speech of the target object, the global timbre feature is added to the first speech feature and input into the duration predictor, the pronunciation duration and semantic representation (pronunciation duration feature) of each phoneme are predicted by upsampling, the pronunciation duration of each phoneme is normalized, and the second speech feature of each phoneme is obtained and output. Exemplarily, assuming that the phoneme sequence is acdb, and the predicted pronunciation durations are 2 1 34 respectively, the pronunciation duration is normalized to aacdddbbbb, and by taking the pronunciation duration feature of each phoneme into account, the pronunciation feature of the target object can be obtained more accurately.

[0075] S25. Input the second speech feature of each phoneme into the decoding module of the acoustic model for decoding processing to obtain the continuous speech feature of each phoneme.

[0076] Reference Figure 3 As shown, the second speech feature is input into the decoding module, and multi-stage step-by-step downsampling is performed to obtain multi-stage continuous semantic features (continuous speech features), and features of different granularities in the speech can be obtained.

[0077] S26, inputting the continuous speech feature of each phoneme into a vocoder model for speech synthesis, and outputting the cloned speech of the target object for the text to be processed.

[0078] Reference Figure 3 As shown, the vocoder model includes: a multi-head vector quantizer and a HifiGAN vocoder; the continuous speech feature of each phoneme is input into the vocoder model for speech synthesis, and the cloned speech of the target object for the text to be processed is output, including:

[0079] The continuous speech features of each phoneme are input into the multi-head vector quantizer for quantization processing to obtain the discrete speech features of each phoneme; the discrete speech features of each phoneme are input into the HifiGAN vocoder for speech synthesis to obtain and output the cloned speech of the target object for the text to be processed.

[0080] The multi-head vector quantizer (multi-head VQ) includes an encoder, a decoder, and a discrete codebook. The discrete codebook replaces continuous features with discrete features, which can facilitate further extraction of the naturalness of speech features, that is, it can make the cloned speech more natural. In the process of discretizing continuous speech features using a multi-head vector quantizer, the problem of lower compression rate and higher model training complexity can be solved.

[0081] HifiGAN vocoder is an efficient vocoder with multi-scale and multi-cycle discriminators. It uses Generative Adversial Networks (GAN) as the basic generation model. HiFiGAN also has a multi-scale discriminator (MSD) and a multi-cycle discriminator, which can maximize the ability of the GAN discriminator to distinguish synthetic or real audio. The generator of HiFiGAN mainly consists of two parts: one is the upsampling structure, which is specifically composed of one-dimensional transposed convolution; the other is the so-called Multi-Receptive Field Fusion (MRF) module, which is mainly responsible for optimizing the sampling points obtained by upsampling, and is specifically composed of a residual network. Discrete speech features are used to generate the final speech through HifiGAN.

[0082] Reference Figure 3 As shown, in the process of training the vocoder model, a multi-order encoder is used to extract the Mel spectrum features corresponding to the target object's voice, and the Mel spectrum features are input into a multi-head vector quantizer for training to improve the similarity between the cloned voice and the target object's voice.

[0083] In the embodiment of the present application, the voice of the target object described in the acoustic model and the vocoder model may be the same voice uttered by the target object, or may be different voices, which is not specifically limited in the embodiment of the present application.

[0084] The embodiment of the present application provides a voice cloning method, comprising: obtaining a phoneme identifier corresponding to a text to be processed, a timbre identifier of a target object, and a mel spectrum corresponding to the voice of the target object; inputting the phoneme identifier, the timbre identifier, and the mel spectrum into an acoustic model, and extracting the phoneme features of the text to be processed and the timbre features of the target object by using an encoding module in the acoustic model; calculating the phoneme features and the timbre features by using a multi-head attention mechanism to obtain a first voice feature of each phoneme in the text to be processed; predicting the pronunciation duration of each phoneme according to the first voice feature of each phoneme and the phoneme feature of the text to be processed to obtain a second voice feature of each phoneme, wherein the second voice feature includes the pronunciation duration of each phoneme; inputting the second voice feature of each phoneme into a decoding module of the acoustic model for decoding processing to obtain a continuous voice feature of each phoneme; inputting the continuous voice feature of each phoneme into a vocoder model for voice synthesis, and outputting a cloned voice of the target object for the text to be processed. In the embodiment of the present application, a multi-head attention mechanism is used to calculate the phoneme features of the text to be processed and the timbre features of the target object to obtain a first speech feature, and a second speech feature is obtained by performing duration prediction based on the first speech feature and the phoneme features of the text to be processed, that is, the second speech feature includes the phoneme features, timbre features, and duration features of the phonemes, which is equivalent to extracting the speech features of the target object for each phoneme from multiple angles, so that the speech synthesized by the vocoder model based on the continuous speech features of each phoneme is closer to the pronunciation of the target object, thereby improving the authenticity of the cloned speech.

[0085] Based on the same inventive concept, as an implementation of the above method, the embodiment of the present application further provides a voice cloning device for executing the above embodiment. The device embodiment corresponds to the above method embodiment. For ease of reading, the device embodiment will not repeat the details of the above method embodiment one by one, but it should be clear that the voice cloning device in this embodiment can correspondingly implement all the contents in the above method embodiment.

[0086] Figure 4 A schematic diagram of the structure of the voice cloning device provided in the embodiment of the present application is shown in FIG. Figure 4 As shown, the voice cloning device 400 provided in this embodiment includes:

[0087] An acquisition module 410 is used to acquire a phoneme identifier corresponding to the text to be processed, a timbre identifier of a target object, and a Mel spectrum corresponding to the speech of the target object;

[0088] The encoding module 420 is used to input the phoneme identifier, the timbre identifier, and the Mel spectrum into the acoustic model, and use the encoding module in the acoustic model to extract the phoneme features of the text to be processed and the timbre features of the target object;

[0089] A calculation module 430 is used to calculate the phoneme feature and the timbre feature using a multi-head attention mechanism to obtain a first speech feature of each phoneme in the text to be processed;

[0090] A processing module 440 is used to predict the pronunciation duration of each phoneme according to the first voice feature of each phoneme and the phoneme feature of the text to be processed, so as to obtain a second voice feature of each phoneme, wherein the second voice feature includes the pronunciation duration of each phoneme;

[0091] A decoding module 450, configured to input the second speech feature of each phoneme into the decoding module of the acoustic model for decoding processing to obtain a continuous speech feature of each phoneme;

[0092] The synthesis module 460 is used to input the continuous speech features of each phoneme into the vocoder model for speech synthesis, and output the cloned speech of the target object for the text to be processed.

[0093] As an optional implementation of the embodiment of the present application, the encoding module 420 is specifically used to input the phoneme identifier corresponding to the text to be processed and the timbre identifier of the target object into the phoneme encoder for feature extraction to obtain the phoneme features of the text to be processed; and input the Mel spectrum corresponding to the speech of the target object into the spectrum encoder for feature extraction to obtain the timbre features of the target object.

[0094] As an optional implementation of an embodiment of the present application, the acoustic model includes: an attention module; the calculation module 430 is specifically used to input the phoneme features of the text to be processed as query content, and the timbre features of the target object as keywords and content values ​​into the attention module; use a multi-head attention mechanism to calculate the phoneme features of the text to be processed and the timbre features of the target object to obtain the first speech features of each phoneme in the text to be processed.

[0095] As an optional implementation of an embodiment of the present application, the acoustic model includes: a duration predictor; the processing module 440 is specifically used to extract the global timbre features of the Mel spectrum; the features obtained by adding the first speech features of each phoneme in the text to be processed and the global timbre features, and the phoneme features of the text to be processed are input into the duration predictor to predict the pronunciation duration of each phoneme, and the pronunciation duration of each phoneme is normalized to obtain and output the second speech features of each phoneme.

[0096] As an optional implementation of an embodiment of the present application, the vocoder model includes: a multi-head vector quantizer and a HifiGAN vocoder; the synthesis module 460 is specifically used to input the continuous speech features of each phoneme into the multi-head vector quantizer for quantization processing to obtain the discrete speech features of each phoneme; input the discrete speech features of each phoneme into the HifiGAN vocoder for speech synthesis, obtain the cloned speech of the target object for the text to be processed and output it.

[0097] As an optional implementation of the embodiment of the present application, the acquisition module 410 is specifically used to acquire the text to be processed, convert the characters in the text to be processed into text phonemes; based on the phoneme identifier dictionary, convert the text phonemes into phoneme identifiers corresponding to the text to be processed.

[0098] As an optional implementation of the embodiment of the present application, the acquisition module 410 is specifically used to acquire the text to be processed. If the text to be processed includes Chinese characters and English characters, the Chinese characters and the English characters are separated, and rhythmic levels are added to the Chinese characters; the Chinese characters and the rhythmic levels are converted into Chinese phonemes, the Chinese phonemes include character phonemes corresponding to the Chinese characters and rhythmic phonemes corresponding to the rhythmic levels, and the English characters are converted into English phonemes; the Chinese phonemes and the English phonemes are concatenated to form text phonemes corresponding to the text to be processed.

[0099] The voice cloning device provided in this embodiment can execute the voice cloning method provided in the above method embodiment, and its implementation principle and technical effect are similar, which will not be repeated here. Each module in the above voice cloning device can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0100] In one embodiment, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of any one of the voice cloning methods described in the above method embodiments when executing the computer program.

[0101] For example, Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 5 As shown, the electronic device provided in this embodiment includes: a memory 51 and a processor 52, the memory 51 is used to store a computer program; the processor 52 is used to execute the steps of the voice cloning method provided in the above method embodiment when calling the computer program, and its implementation principle and technical effect are similar, which will not be repeated here. Those skilled in the art can understand that Figure 5 The structure shown in the figure is merely a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the electronic device to which the scheme of the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0102] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of any one of the voice cloning methods described in the above method embodiments are implemented.

[0103] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in various forms, such as static random access memory (SRAM) and dynamic random access memory (DRAM).

[0104] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0105] The above description is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A voice cloning method, characterized in that: include: Obtaining a phoneme identifier corresponding to the text to be processed, a timbre identifier of a target object, and a Mel spectrum corresponding to the speech of the target object; Input the phoneme identifier, the timbre identifier, and the Mel spectrum into an acoustic model, and use an encoding module in the acoustic model to extract the phoneme features of the text to be processed and the timbre features of the target object; The phoneme feature and the timbre feature are calculated using a multi-head attention mechanism to obtain a first speech feature of each phoneme in the text to be processed; Predicting the pronunciation duration of each phoneme according to the first voice feature of each phoneme and the phoneme feature of the text to be processed, to obtain a second voice feature of each phoneme, wherein the second voice feature includes the pronunciation duration of each phoneme; Inputting the second speech feature of each phoneme into the decoding module of the acoustic model for decoding processing to obtain the continuous speech feature of each phoneme; Inputting the continuous speech feature of each phoneme into a vocoder model for speech synthesis, and outputting the cloned speech of the target object for the text to be processed; The encoding module of the acoustic model includes a phoneme encoder and a spectrum encoder; the phoneme identifier, the timbre identifier, and the Mel spectrum are input into the acoustic model, and the phoneme features of the text to be processed and the timbre features of the target object are extracted using the encoding module of the acoustic model, including: inputting the phoneme identifier corresponding to the text to be processed and the timbre identifier of the target object into the phoneme encoder for feature extraction to obtain the phoneme features of the text to be processed; inputting the Mel spectrum corresponding to the speech of the target object into the spectrum encoder for feature extraction to obtain the timbre features of the target object; The acoustic model includes: a duration predictor; predicting the pronunciation duration of each phoneme according to the first speech feature of each phoneme and the phoneme feature of the text to be processed to obtain the second speech feature of each phoneme, including: extracting the global timbre feature of the Mel spectrum; inputting the feature obtained by adding the first speech feature of each phoneme in the text to be processed and the global timbre feature, and the phoneme feature of the text to be processed into the duration predictor, predicting the pronunciation duration of each phoneme, and normalizing the pronunciation duration of each phoneme to obtain and output the second speech feature of each phoneme.

2. The method according to claim 1, characterized in that The acoustic model includes: an attention module; the multi-head attention mechanism is used to calculate the phoneme feature and the timbre feature to obtain the first speech feature of each phoneme in the text to be processed, including: Input the phoneme features of the text to be processed as query content and the timbre features of the target object as keywords and content values ​​into the attention module; The phoneme features of the text to be processed and the timbre features of the target object are calculated using a multi-head attention mechanism to obtain the first speech feature of each phoneme in the text to be processed.

3. The method according to claim 1, characterized in that The vocoder model includes: a multi-head vector quantizer and a HifiGAN vocoder; the continuous speech feature of each phoneme is input into the vocoder model for speech synthesis, and the cloned speech of the target object for the text to be processed is output, including: Inputting the continuous speech feature of each phoneme into the multi-head vector quantizer for quantization processing to obtain the discrete speech feature of each phoneme; The discrete speech features of each phoneme are input into the HifiGAN vocoder for speech synthesis, and the cloned speech of the target object for the text to be processed is obtained and output.

4. The method according to any one of claims 1 to 3, characterized in that: The step of obtaining the phoneme identifier corresponding to the text to be processed includes: Acquire a text to be processed, and convert characters in the text to be processed into text phonemes; Based on a phoneme identifier dictionary, the text phonemes are converted into phoneme identifiers corresponding to the text to be processed.

5. The method according to claim 4, characterized in that The step of obtaining the text to be processed and converting characters in the text to be processed into text phonemes includes: Acquire a text to be processed, and if the text to be processed includes Chinese characters and English characters, separate the Chinese characters from the English characters, and add a rhythm level to the Chinese characters; Converting the Chinese characters and the prosody levels into Chinese phonemes, wherein the Chinese phonemes include character phonemes corresponding to the Chinese characters and prosody phonemes corresponding to the prosody levels, and converting the English characters into English phonemes; The Chinese phonemes and the English phonemes are concatenated to form text phonemes corresponding to the text to be processed.

6. A voice cloning device, characterized in that: include: An acquisition module, used to acquire a phoneme identifier corresponding to the text to be processed, a timbre identifier of a target object, and a Mel spectrum corresponding to the speech of the target object; An encoding module, used for inputting the phoneme identifier, the timbre identifier, and the Mel spectrum into an acoustic model, and extracting the phoneme features of the text to be processed and the timbre features of the target object by using the encoding module in the acoustic model; A calculation module, used for calculating the phoneme feature and the timbre feature by using a multi-head attention mechanism to obtain a first speech feature of each phoneme in the text to be processed; A processing module, configured to predict the pronunciation duration of each phoneme according to the first voice feature of each phoneme and the phoneme feature of the text to be processed, so as to obtain a second voice feature of each phoneme, wherein the second voice feature includes the pronunciation duration of each phoneme; A decoding module, used for inputting the second speech feature of each phoneme into the decoding module of the acoustic model for decoding processing to obtain the continuous speech feature of each phoneme; A synthesis module, used for inputting the continuous speech feature of each phoneme into a vocoder model for speech synthesis, and outputting the cloned speech of the target object for the text to be processed; The encoding module of the acoustic model includes a phoneme encoder and a spectrum encoder; the encoding module is specifically used to input the phoneme identifier corresponding to the text to be processed and the timbre identifier of the target object into the phoneme encoder for feature extraction to obtain the phoneme features of the text to be processed; input the Mel spectrum corresponding to the speech of the target object into the spectrum encoder for feature extraction to obtain the timbre features of the target object; The acoustic model includes: a duration predictor; the processing module is specifically used to extract the global timbre feature of the Mel spectrum; the feature obtained by adding the first speech feature of each phoneme in the text to be processed and the global timbre feature, and the phoneme feature of the text to be processed are input into the duration predictor to predict the pronunciation duration of each phoneme, and the pronunciation duration of each phoneme is normalized to obtain and output the second speech feature of each phoneme.

7. An electronic device comprising: A memory and a processor, wherein the memory stores a computer program, and wherein the processor implements the voice cloning method according to any one of claims 1 to 5 when executing the computer program.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the voice cloning method according to any one of claims 1 to 5 is implemented.

Images