A voice conversion method based on a vocoder, a storage medium and an electronic device

Abstract

The application relates to the technical field of deep learning and natural language processing, and discloses a voice conversion method based on a vocoder, a storage medium and an electronic device, which comprises the following steps: constructing an any-to-one voice conversion model, performing content coding on source voice audio, obtaining acoustic characteristics through an acoustic model, obtaining converted voice through a vocoder, extracting the mean and variance of a target speaker, inputting the spectrum of the converted voice obtained through the vocoder into a feature extractor composed of a convolution, a WaveNet module and a linear radiation transformation to obtain features Z_source, converting Z_source into Z_target by using the mean and variance of the target speaker, inputting the features Z_source into a UnivNet structure to obtain converted voice, the method not only has the function of the vocoder, but also can well convert voice input of any kind of language or dialect to a target timbre, and well retains other features in the original voice except the speaker and the content.

Description

Technical Field

[0001] This application relates to the fields of deep learning and natural language processing technology, and in particular to a vocoder-based speech conversion method, storage medium, and electronic device. Background Technology

[0002] With the widespread application of deep learning in various fields, many tasks in the field of speech have also made great progress, such as speech synthesis and speech conversion.

[0003] Speech conversion is the process of converting one person's speech into another person's voice while preserving its content. Speech conversion types can be broadly categorized based on whether the speaker is in the training set: any-to-one, many-to-many, any-to-many, and any-to-any. "Any" means the input can be any person, "one" means a specific person, and "many" means a limited number of people. An "any-to-one" model can convert any speaker into a specific person. Therefore, it cannot convert to other speakers. "Many-to-many" generally means that people in the training data can convert to each other, but people outside the training set cannot be converted. "Any-to-many" has no restrictions on the input speaker, but the target speaker can only appear in the training set. "Any-to-any" is the most difficult of these, capable of converting any person's voice into any other person's voice. The general structure of a speech conversion model is: 1. Content Encoder: Encodes the input speech to obtain the content of the speech and removes the speaker's information. 2. Speaker Encoder: Also encodes speech, but captures the speaker's information while removing the content information. 3. Decoder: Decodes the outputs of the content encoder and speaker encoder, outputting specific acoustic feature information or a speech waveform.

[0004] Speech cloning involves fine-tuning a target speaker's voice based on a single sentence or a few sentences from the target speaker, learning the target speaker's vocal characteristics to enable the target speaker to speak any content. Both speech cloning and speech conversion require a single sentence or a few sentences from the target speaker as a reference to learn their vocal timbre. The difference lies in that speech cloning uses arbitrary text as input, while speech conversion uses the source speaker's voice.

[0005] Current speech conversion technologies follow these approaches: 1. Existing vocoders can only convert a person's acoustic features into their speech waveform, preserving the speaker and the content of their speech during the conversion process. 2. Current speech conversion systems require encoding the input speech to remove speaker information, and simultaneously encoding the target voice to extract speaker information. However, speaker information lacks explicit features for removal and extraction; only a vector of latent variables is extracted. The performance is significantly worse for extracting speakers from outside the training set.

[0006] The existing speech cloning technology approach involves: 1. Training a good speech synthesis model. 2. Using the trained speech synthesis model to retrain and fine-tune the target speaker, learning the target speaker's timbre. In practical product applications, this retraining requires user waiting time and places high demands on the deployed equipment. Summary of the Invention

[0007] The purpose of this application is to overcome the shortcomings of the prior art and provide a voice coder-based speech conversion method, storage medium, and electronic device.

[0008] Firstly, a vocoder-based speech conversion method is provided, including:

[0009] Acquire the raw speech, the target speaker's speech, and the speech dataset;

[0010] Construct an any-to-one speech conversion model and train it.

[0011] The original speech is input into an any-to-one speech conversion model and converted into the target middleman speech;

[0012] All the speech in the speech dataset is input into an any-to-one speech conversion model to convert it into human-to-human speech in order to build a parallel dataset;

[0013] Construct a speaker coding structure and use the speaker coding structure to extract the vector of the target speaker, where the vector includes the mean and variance;

[0014] Construct a feature extractor and input the spectrum of the target man-in-the-middle speech into the feature extractor to obtain the feature Z_source;

[0015] The feature Z_source is transformed into the feature Z_target using the target speaker's vector;

[0016] The feature Z_target is input into the vocoder to obtain the converted target speech.

[0017] Furthermore, the any-to-one speech conversion model includes a content encoder, an acoustic model, and a vocoder. The content encoder is used to acquire the content of the input speech and remove the speaker's information. The acoustic model is used to extract acoustic features. The vocoder is used to convert the outputs of the content encoder and the acoustic model into the speech of a specified middleman.

[0018] Furthermore, the original speech is input into an any-to-one speech conversion model and converted into the target middleman speech, including the following steps:

[0019] The original speech is input into the content encoder to perform content encoding on the original speech in order to remove the original speaker information from the original speech;

[0020] The original speech is processed through an acoustic model to obtain its acoustic features.

[0021] The acoustic features of the original speech are input into a vocoder to obtain the target man-in-the-middle speech.

[0022] Furthermore, the speech data in the speech dataset is input into an any-to-one speech conversion model to convert it into human-to-the-middle speech, including the following steps:

[0023] Each voice record in the voice dataset is input into the content encoder to encode the content of the voice;

[0024] Each speech item in the speech dataset is processed through an acoustic model to obtain the acoustic features of the speech.

[0025] The acoustic features of the speech in the above speech dataset are input one by one into the vocoder to obtain the man-in-the-middle speech corresponding to the speech in the speech dataset.

[0026] Furthermore, the speech dataset contains speech data from multiple different speakers.

[0027] Furthermore, the speaker coding structure includes a conformer network model and an ECAPA-TDNN model, wherein the conformer network model is used to extract speaker features, and the ECAPA-TDNN model is used for speaker recognition.

[0028] Furthermore, the feature extractor includes a convolution module, a WaveNet module, and a linear radiometric transformation module.

[0029] Furthermore, the feature Z_source is transformed into the feature Z_target using the target speaker's vector, including:

[0030] The target middleman speech is denoted as source, and the speech parallel to the target middleman speech is denoted as target. The (source, target) pairs are input into the vocoder structure. After passing through the feature extraction network, the source and target are processed separately, and then passed through the common network again. During training, the target features obtained after separate processing are used to guide the learning of source features. Here, Z_target = InstanceNorm(Z_source)*Std+Mean, where Std represents the variance and Mean represents the mean.

[0031] In a second aspect, a computer-readable storage medium is provided that stores program code for execution by a device, the program code including steps for performing a method as described in any implementation of the first aspect.

[0032] Thirdly, an electronic device is provided, the electronic device including a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the method as in any of the implementations of the first aspect.

[0033] This application has the following beneficial effects: the method not only has the function of a vocoder, but also can convert speech input of any language or dialect into the target timbre very well, and preserves other features of the original speech other than the speaker and content. In addition, while optimizing the process of speech cloning, it only requires one sentence of the target speaker's speech and does not require online training, thus improving the practicality of speech cloning. Attached Figure Description

[0034] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments of this application and their descriptions are used to explain this application and do not constitute an undue limitation of this application.

[0035] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1 This is a flowchart of a vocoder-based speech conversion method according to Embodiment 1 of this application. Detailed Implementation

[0037] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0038] Example 1

[0039] An embodiment of this application discloses a vocoder-based speech conversion method, comprising: acquiring original speech, target speaker speech, and a speech dataset; constructing an any-to-one speech conversion model and training the any-to-one speech conversion model; inputting the original speech into the any-to-one speech conversion model to convert it into target intermediary speech; inputting all speech from the speech dataset into the any-to-one speech conversion model to convert it into intermediary speech to construct a parallel dataset; constructing a speaker coding structure and using the speaker coding structure to extract the target speaker's vector, wherein the vector includes the mean and variance; constructing a feature extractor and extracting the target intermediary speech. The spectrum of human speech is input into a feature extractor to obtain feature Z_source; feature Z_source is converted into feature Z_target using the target speaker's vector; feature Z_target is input into a vocoder to obtain the converted target speech. This method not only has the function of a vocoder, but also can convert speech input of any language or dialect into the target timbre very well. It also preserves other features in the original speech besides the speaker and content very well. In addition, the speech cloning process has been optimized, requiring only one sentence of the target speaker's speech, and online training is not required, which improves the practicality of speech cloning.

[0040] Specifically, Figure 1 A flowchart of the vocoder-based speech conversion method in Embodiment 1 of the application is shown, including:

[0041] S101. Obtain the original speech, the target speaker's speech, and the speech dataset;

[0042] In this application, the original speech is the speech input by the user that needs to be converted, and the target speaker's speech is the speech input by the user that needs to be converted from the original speech to the target speaker's speech. For example, the original speech is a speech spoken by A, and the target speaker's speech is a speech spoken by B. The content of the two speech pieces may be different. The purpose of this method is to convert the original speech into speech spoken by B, that is, to change the speaker in the original speech from A to B, without changing the content of the speech or the background sound. For example, the total duration of the speech dataset may be 4,000 hours, 5,000 hours, or 6,000 hours, etc. The specific total duration of the speech can be adjusted according to the needs. These speech pieces come from multiple different speakers, and the number of speech pieces may be eight million, nine million, or ten million, etc. It should be noted that the speech in this application should not be understood as only the speech sent or received in communication software. Recordings, film and television audio, songs, operas, and other electronic audio can all be regarded as the speech in this application.

[0043] S102. Construct an any-to-one speech conversion model and train the any-to-one speech conversion model;

[0044] The any-to-one speech conversion model is trained based on a soft-VC model. Specifically, the any-to-one speech conversion model includes a content encoder, an acoustic model, and a vocoder. The content encoder is used to obtain the content of the input speech and remove the speaker information. The acoustic model is used to extract acoustic features. The vocoder is used to convert the output of the content encoder and the acoustic model into the speech of a specified middleman, that is, it can convert the voice of any person into the timbre of a specified middleman.

[0045] It should be noted that the function of the vocoder is:

[0046] 1. The spectrum of the original speech is input into a feature extractor consisting of a convolutional WaveNet module and a linear eccentric transformation to obtain feature Z;

[0047] 2. By inputting the feature Z into the UnivNet structure, the original speech can be obtained again, which is to say, the original speech can be reconstructed.

[0048] S103. Input the original speech into the any-to-one speech conversion model and convert it into the target middleman speech;

[0049] Specifically, the process of inputting the original speech into an any-to-one speech conversion model and converting it into the target middleman speech includes the following steps:

[0050] The original speech is input into the content encoder to encode the original speech content in order to remove the original speaker information from the original speech. The content encoder can encode the input speech to obtain the content of the speech and remove the speaker information.

[0051] The original speech is processed through an acoustic model to obtain the acoustic features of the original speech. The acoustic model is able to extract acoustic features from the input speech.

[0052] The acoustic features of the original speech are input into a vocoder to obtain the target middleman speech. The vocoder can convert the acoustic features of any person into the speech waveform of the same person, while keeping the content of the speech unchanged during the conversion process.

[0053] It should be noted that in this step, the original speech is input into the trained any-to-one speech conversion model and converted into the speech of the specified target middleman. For example, the middleman is Li Si. Regardless of whether the original speech is spoken by Li Si or Wu Liu, the original speech will be converted into the speech of Li Si. The content of the speech and the background sound are not changed. Only the timbre of the speaker is changed, which completes the speech conversion of the any-to-one part.

[0054] S104. Input all the speech in the speech dataset into the any-to-one speech conversion model to convert it into human-to-human speech in order to build a parallel dataset;

[0055] Specifically, the speech data in the speech dataset is input into an any-to-one speech conversion model to convert it into human-to-human speech, including the following steps:

[0056] The speech data in the speech dataset is input into the content encoder one by one to encode the speech content. The content encoder can encode the input speech, obtain the content of the speech, and remove the speaker information.

[0057] The speech data in the speech dataset is processed one by one through an acoustic model to obtain the acoustic features of the speech. The acoustic model is able to extract acoustic features from the input speech.

[0058] The acoustic features of each speech in the above speech dataset are input into a vocoder to obtain the man-in-the-middle speech corresponding to the speech in the speech dataset. That is, each speech in the speech dataset has a corresponding converted speech, thus constructing a parallel dataset that is parallel to the speech dataset.

[0059] S105. Construct a speaker coding structure and use the speaker coding structure to extract the target speaker's vector, where the vector includes the mean and variance;

[0060] Specifically, the speaker encoding structure includes a conformer network model and an ECAPA-TDNN model. The conformer network model is used to extract speaker features, and the ECAPA-TDNN model is used for speaker recognition. When extracting vectors, the extracted speech spectrogram features are input into the conformer network model to obtain speaker features with the same shape and size. Then, the speaker features are input into the ECAPA-TDNN model to construct a speaker distribution and obtain the speaker's mean and variance. It should be noted that different sentences spoken by the same speaker should fall into the same distribution.

[0061] S106. Construct a feature extractor and input the spectrum of the target man-in-the-middle speech into the feature extractor to obtain the feature Z_source;

[0062] Specifically, the feature extractor includes a convolution module, a WaveNet module, and a linear eccentric transformation module. The results of steps S104 and S105 are used here. The spectrum of the speech is input into the feature extractor composed of the convolution, WaveNet module, and linear eccentric transformation to obtain feature Z. The target middleman speech is denoted as source. Z_source and Z_target refer to the feature Z of the target middleman speech and the feature Z of the target speaker speech, respectively.

[0063] S107. Transform the feature Z_source into the feature Z_target using the target speaker's vector;

[0064] Specifically, Z_source and Z_target refer to the features Z of the target middleman speech and the target speaker speech, respectively. The (source, target) pairs are input into the vocoder structure. After passing through the feature extraction network, the source and target are processed separately, and then passed through a shared network again. During training, the target features obtained after separate processing are used to guide the learning of the source features. Here, Z_target = InstanceNorm(Z_source) * Std + Mean, where Std represents the variance (here referring to the variance of the target speaker speech), and Mean represents the mean (here referring to the mean of the target speaker speech). That is, the feature Z_source is normalized by InstanceNorm to remove the speaker features of the original speech, and then multiplied by the variance and mean of the target speaker, thus transforming Z_source into Z_target. This allows for supervised guidance in the generation of Z_target. The implementation is simple based on the previous points; Z_source can be transformed into Z_target, and there are real Z_targets on the parallel dataset mentioned earlier. See details. Figure 1 .

[0065] S108. Input the feature Z_target into the vocoder to obtain the converted target speech, wherein the converted speech retains the speech content of the original speech and also has the timbre of the target speaker.

[0066] It should be noted that UnivNet is used as the vocoder here. It can convert features Z into high-quality speech and is robust to both the speaker and the speaker's language. Of course, other neural network models (such as WaveNet, Parallel WaveNet, or Hifi-Gan) can also be used as vocoders to predict the real speech waveform.

[0067] In summary, this method can effectively reconstruct speech from the acoustic features of any person using a single sentence. It decomposes any-to-any speech conversion into two parts: any-to-one and one-to-any, thereby reducing the difficulty of any-to-any conversion and improving the conversion effect. In terms of speech cloning: using a trained speech synthesis model, speech cloning can be implemented in two parts, namely speech synthesis plus a one-sentence speech conversion model. This only requires the target speaker to provide a sentence, and in actual product applications, no additional retraining time is required for users to wait.

[0068] Example 2

[0069] The second embodiment of this application relates to a computer-readable storage medium that stores program code for execution by a device, the program code including steps for performing the method in any implementation of the first embodiment of this application;

[0070] The computer-readable storage medium may be a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM); the computer-readable storage medium may store program code, and when the program stored in the computer-readable storage medium is executed by a processor, the processor is used to perform the steps of the method in any of the implementations of Embodiment 1 of this application.

[0071] Example 3

[0072] The electronic device involved in Embodiment 3 of this application includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor. When the program or instructions are executed by the processor, they implement the method in any of the implementations in Embodiment 1 of this application.

[0073] The processor can be a general-purpose central processing unit (CPU), microprocessor, application-specific integrated circuit (ASIC), graphics processing unit (GPU), or one or more integrated circuits, used to execute relevant programs to implement the method in any of the implementations of Embodiment 1 of this application.

[0074] The processor can also be an integrated circuit electronic device with signal processing capabilities. In implementation, each step of the method in any of the implementations of Embodiment 1 of this application can be completed by the integrated logic circuitry in the processor's hardware or by software instructions.

[0075] The aforementioned processor can also be a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory; the processor reads information from the memory and, in conjunction with its hardware, completes the functions required by the units included in the data processing apparatus of the embodiments of this application, or executes the methods in any implementation of Embodiment 1 of this application.

[0076] The above are merely preferred embodiments of this application; however, the scope of protection of this application is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in this application, based on the technical solution and its improved concept, should be covered within the scope of protection of this application.

Claims

1. A speech conversion method based on a vocoder, characterized in that, include: Acquire the raw speech, the target speaker's speech, and the speech dataset; Construct an any-to-one speech conversion model and train it. The original speech is input into an any-to-one speech conversion model and converted into the target middleman speech; All the speech in the speech dataset is input into an any-to-one speech conversion model to convert it into human-to-human speech in order to build a parallel dataset; Construct a speaker coding structure and use the speaker coding structure to extract the vector of the target speaker, where the vector includes the mean and variance; A feature extractor is constructed, which includes a convolution module, a WaveNet module, and a linear eccentric transformation module. The spectrum of the target man-in-the-middle speech is input into the feature extractor to obtain the feature Z_source. The feature Z_source is transformed into the feature Z_target using the target speaker's vector; The feature Z_target is input into the vocoder to obtain the converted target speech.

2. The speech conversion method based on a vocoder according to claim 1, characterized in that, The any-to-one speech conversion model includes a content encoder, an acoustic model, and a vocoder. The content encoder is used to acquire the content of the input speech and remove the speaker information. The acoustic model is used to extract acoustic features. The vocoder is used to convert the output of the content encoder and the acoustic model into the speech of the specified middleman.

3. The speech conversion method based on a vocoder according to claim 2, characterized in that, The original speech is input into an any-to-one speech conversion model and converted into the target human-in-the-middle speech, including the following steps: The original speech is input into the content encoder to perform content encoding on the original speech in order to remove the original speaker information from the original speech; The original speech is processed through an acoustic model to obtain its acoustic features. The acoustic features of the original speech are input into a vocoder to obtain the target man-in-the-middle speech.

4. The speech conversion method based on a vocoder according to claim 2, characterized in that, The speech data in the speech dataset is input into an any-to-one speech conversion model to convert it into human-to-human speech, including the following steps: Each voice record in the voice dataset is input into the content encoder to encode the content of the voice; Each speech item in the speech dataset is processed through an acoustic model to obtain the acoustic features of the speech. The acoustic features of the speech in the above speech dataset are input one by one into the vocoder to obtain the man-in-the-middle speech corresponding to the speech in the speech dataset.

5. The speech conversion method based on a vocoder according to claim 1, characterized in that, The speech dataset contains speech data from multiple different speakers.

6. The speech conversion method based on a vocoder according to claim 1, characterized in that, The speaker coding structure includes a conformer network model and an ECAPA-TDNN model, wherein the conformer network model is used to extract speaker features, and the ECAPA-TDNN model is used for speaker recognition.

7. The speech conversion method based on a vocoder according to claim 1, characterized in that, The feature Z_source is transformed into the feature Z_target using the target speaker's vector, including: The target middleman speech is denoted as source, and the speech parallel to the target middleman speech is denoted as target. The (source, target) pairs are input into the vocoder structure. After passing through the feature extraction network, the source and target are processed separately, and then passed through the common network again. During the training process, the target features obtained after separate processing are used to guide the learning of source features. Here, Z_target = InstanceNorm(Z_source) * Std + Mean, where Std refers to the variance and mean refers to the mean.

8. A computer-readable storage medium, characterized in that, The computer-readable medium stores program code for execution by the device, the program code including steps for performing the method as described in any one of claims 1-7.

9. An electronic device, characterized in that, The electronic device includes a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the method as described in any one of claims 1-7.

Images