Voice conversion model training method, voice conversion method, electronic equipment and storage medium
Through the adversarial training voice conversion model, combined with content encoder and vocal encoder, the problems of high voice conversion efficiency and cost in the prior art are solved, and a more efficient personalized voice conversion effect is achieved.
Patent Information
- Application Number
- CN202410976926.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-18
- Publication Date
- 2024-11-08
AI Technical Summary
Existing speech conversion technologies have shortcomings in terms of generation efficiency and machine learning training costs, especially in terms of limited effectiveness in personalized speech conversion.
Using a speech conversion model including content encoder, vocal encoder and waveform generator, the model is trained through an adversarial training method, and the target audio is generated using pre-trained content encoder and vocal encoder, reducing the vocal coder's vocal coder's The dependency of the company improves the generation efficiency and reduces the training cost.
It improves the generation efficiency of speech conversion, reduces the cost of machine learning training, and improves the accuracy and nature of personalized speech.
Smart Images

Figure CN118918909A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of multimedia content processing, and in particular to a speech conversion model training method, a speech conversion method, an electronic device and a storage medium. Background Art
[0002] Voice Conversion (VC) technology is widely used in voice assistants, chatbots, audiobooks, virtual humans, etc. With the growing demand for personalized voice, such as the demand for voice conversion to the voice of a specific anchor or celebrity, the importance of this technology is becoming increasingly significant.
[0003] Current speech conversion usually consists of two main parts: acoustic model (AM) and vocoder. The acoustic model is responsible for outputting Mel spectrum according to the input content audio and target timbre audio, while the vocoder synthesizes the corresponding audio waveform according to the Mel spectrum.
[0004] However, current voice conversion technologies still have some limitations. For example, the involvement of vocoders in traditional voice conversion schemes leads to reduced generation efficiency and a significant increase in machine learning training costs. In addition, these schemes are often limited in their effectiveness in accurately replicating specific timbres, especially when faced with the need for personalized voice conversion.
[0005] In view of the above limitations, we hope to explore more efficient and low-cost solutions to improve the generation efficiency of speech conversion, reduce the cost of machine learning training, and improve the accuracy and naturalness of personalized speech.
[0006] The content of this background technology description is only for facilitating understanding of the relevant technology in this field and is not regarded as an admission of the prior art. Summary of the invention
[0007] Therefore, the embodiments of the present application intend to provide a speech conversion model training method, a speech conversion method, and related electronic devices and storage media. Through the solutions of the embodiments of the present application, the generation efficiency of speech conversion can be improved, the cost of machine learning training can be reduced, and the accuracy and naturalness of personalized speech can be improved.
[0008] In a first aspect, an embodiment of the present application provides a method for training a speech conversion model, wherein the speech conversion model includes a content encoder, a vocal encoder, and a waveform generator. In this embodiment, the training method includes:
[0009] pre-training the content encoder;
[0010] pre-training the vocal encoder; and
[0011] The waveform generator and the corresponding discriminator of the speech conversion model are adversarially trained, wherein the adversarial training comprises iteratively executing the following steps until a preset iteration termination condition is reached:
[0012] Input the training audio into the content encoder and the voice encoder respectively to output content features and voice features;
[0013] Input the content features and vocal features into the waveform generator to generate training audio,
[0014] The discriminator discriminates the real audio and / or the generated audio from the training audio and the training generated audio;
[0015] updating the parameters of the waveform generator according to a first loss value determined according to a first correlation of the discrimination result;
[0016] The parameters of the discriminator are updated by a second loss value determined according to a second correlation of the discrimination result.
[0017] In the embodiment of the present application, when the adversarial training is iteratively performed, the parameters of the content encoder and the vocal encoder remain fixed.
[0018] In an embodiment of the present application, the adversarial training further includes:
[0019] Inputting the training generated audio into the content encoder to obtain generated content features;
[0020] determining a third loss value according to the content feature and the generated content feature;
[0021] Wherein, updating the parameters of the waveform generator according to the first loss value determined according to the first correlation of the discrimination result includes:
[0022] Parameters of the waveform generator are updated by the first loss value and the third loss value.
[0023] In an embodiment of the present application, the adversarial training further includes:
[0024] Inputting the training generated audio into the vocal encoder to obtain generated vocal features;
[0025] Determine a fourth loss value according to the vocal feature and the generated vocal feature;
[0026] Wherein, updating the parameters of the waveform generator according to the first loss value determined according to the first correlation of the discrimination result includes:
[0027] Parameters of the waveform generator are updated according to the first loss value and the fourth loss value.
[0028] In an embodiment of the present application, the adversarial training further includes:
[0029] Inputting the training generated audio into the content encoder to obtain generated content features;
[0030] determining a third loss value according to the content feature and the generated content feature;
[0031] Inputting the training generated audio into the vocal encoder to obtain generated vocal features;
[0032] Determine a fourth loss value according to the vocal feature and the generated vocal feature;
[0033] Wherein, updating the parameters of the waveform generator according to the first loss value determined according to the first correlation of the discrimination result includes:
[0034] Parameters of the waveform generator are updated by the first loss value, the third loss value, and the fourth loss value.
[0035] In an embodiment of the present application, the content encoder is an attention-enhanced content encoder, comprising: a sampling layer for sampling input audio features to obtain a downsampled feature sequence; one or more multi-head attention modules for acquiring feature associations in the input feature sequence based on attention; and a decoder for decoding the feature sequence to obtain a posterior probability of phonemes for each frame of audio.
[0036] In an embodiment of the present application, the vocal encoder is an attention-enhanced vocal encoder, comprising: a feature extraction layer for extracting input audio features; one or more squeeze-excited residual network blocks for enhancing given vocal features in the extracted input audio features; and an output classification layer for outputting a classification of the given vocal features.
[0037] In an embodiment of the present application, the vocal feature includes a speaker's timbre feature sequence, and / or the content feature includes a phoneme posterior probability.
[0038] In an embodiment of the present application, before the content features and the vocal features are input into the waveform generator, the adversarial training further includes: splicing the content features and the vocal features.
[0039] In a second aspect, an embodiment of the present application provides a voice conversion method, which may include:
[0040] Inputting the target content audio into a content encoder of a speech conversion model to output target content features, wherein the speech conversion model is obtained by executing the method described in any embodiment of the present application;
[0041] Inputting the target vocal audio into the vocal encoder of the speech conversion model to output the target vocal features; and
[0042] The target content features and target vocal features are input into the waveform generator of the speech conversion model to generate target generated audio.
[0043] In the embodiment of the present application, before inputting the target content feature and the target vocal feature into the waveform generator of the speech conversion model, the method further includes:
[0044] The target content feature and the target vocal feature are concatenated.
[0045] In a third aspect, an embodiment of the present application provides a speech conversion model, which includes a content encoder, a voice encoder, and a waveform generator, wherein the speech conversion model is trained by any training method described in any embodiment of the present application.
[0046] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the method of any embodiment of the present application is implemented.
[0047] In a fifth aspect, an embodiment of the present application provides an electronic device, comprising: a processor and a memory storing a computer program, wherein the processor is configured to execute any method of the embodiment of the present application when running the computer program.
[0048] The present technical solution provides a training method for a speech conversion model including a sound encoder, a content encoder and a waveform generator, including a pre-trained content encoder; a pre-trained human voice encoder; and a waveform generator and a corresponding discriminator for adversarially training the speech conversion model. The adversarial training includes iteratively executing the following steps until a preset iteration termination condition is reached: inputting the training audio into the content encoder and the human voice encoder respectively to output content features and human voice features; inputting the content features and human voice features into the waveform generator to generate training generated audio, and the discriminator discriminates the real audio and / or generated audio in the training audio and the training generated audio; updating the parameters of the waveform generator by the first loss value; and updating the parameters of the discriminator by the second loss value. The speech conversion model obtained thereby improves the generation efficiency of speech conversion, reduces the cost of machine learning training, and at the same time improves the accuracy and naturalness of personalized speech.
[0049] Other optional features and technical effects of the embodiments of the present application are partially described below, and partially can be understood by reading this document. BRIEF DESCRIPTION OF THE DRAWINGS
[0050] Hereinafter, embodiments of the present application will be described in detail with reference to the accompanying drawings. The elements shown are not limited by the proportions shown in the accompanying drawings. The same or similar reference numerals in the accompanying drawings represent the same or similar elements, wherein:
[0051] Figure 1 An example flow chart showing a method for training a speech conversion model according to an embodiment of the present application;
[0052] Figure 2 A schematic diagram showing the structure of a content encoder of a speech conversion model according to an embodiment of the present application;
[0053] Figure 3 A schematic diagram showing the structure of a content encoder of a speech conversion model according to an embodiment of the present application;
[0054] Figure 4 A schematic diagram showing the structure of a voice encoder of a speech conversion model according to an embodiment of the present application;
[0055] Figure 5 A schematic diagram showing the structure of a voice encoder of a speech conversion model according to an embodiment of the present application;
[0056] Figure 6 An example flow chart showing a method for training a speech conversion model according to an embodiment of the present application;
[0057] Figure 7 A schematic diagram showing the module structure of adversarial training for implementing the speech conversion model training method of an embodiment of the present application is shown;
[0058] Figure 8 An example flow chart showing a method for training a speech conversion model according to an embodiment of the present application;
[0059] Fig. 9 An example flow chart showing a method for training a speech conversion model according to an embodiment of the present application;
[0060] Fig.10 An example flow chart showing a method for training a speech conversion model according to an embodiment of the present application;
[0061] Fig.11 An example flow chart showing the voice conversion method according to an embodiment of the present application;
[0062] Fig.12 A schematic diagram showing the module structure of the voice conversion method for implementing the embodiment of the present application is shown;
[0063] Fig.13 An exemplary structural diagram showing a speech conversion model training device according to an embodiment of the present application;
[0064] Fig.14 An exemplary structural diagram of a speech conversion model according to an embodiment of the present application is shown;
[0065] Fig.15 An exemplary structural diagram of an electronic device capable of implementing the method according to an embodiment of the present application is shown. DETAILED DESCRIPTION
[0066] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with specific implementation methods and drawings. Here, the illustrative implementation methods and descriptions of the present application are used to explain the present application, but are not intended to limit the present application.
[0067] As used herein, the term "including" and its variations mean open inclusion, i.e., "including but not limited to". Unless otherwise stated, the term "or" means "and / or". The term "based on" means "based at least in part on". The terms "an example embodiment" and "an embodiment" mean "at least one example embodiment". The term "another embodiment" means "at least one additional embodiment". The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below.
[0068] In the embodiments of the present application, voice conversion (VC) generates corresponding voice audio according to the input content audio and the target human voice, such as timbre audio, and has the conventional meaning in the field of natural language processing (NLP).
[0069] In the embodiments of the present application, "timbre" is a basic feature that distinguishes one sound from another, and different people have different timbre.
[0070] In the embodiments of the present application, "model" has the conventional meaning in the field of machine learning. For example, the model can be a machine learning or deep learning model, such as a machine learning or deep learning model including the above-mentioned network or composed of the above-mentioned network.
[0071] In the embodiments of the present application, “loss function” and “loss value” have conventional meanings in the field of machine learning.
[0072] In the embodiment of the present application, a "phoneme" is the smallest unit of speech divided according to the natural properties of speech, and one pronunciation action forms a phoneme. A "phoneme sequence" is a sequence composed of a number of phonemes.
[0073] The embodiments of the present application provide a speech conversion model training method and device, a speech conversion method, a speech conversion model, a storage medium and an electronic device. The method, device / model can be implemented with the aid of one or more computers. In some embodiments, the device / model can be implemented by software, hardware, or a combination of software and hardware. In some embodiments, the electronic device or computer can be implemented by a computer described herein or other electronic device that can implement the corresponding function.
[0074] Traditional voice conversion (VC) technology is usually similar to text-to-speech (TTS) technology, including two parts: acoustic model (AM) and vocoder. The acoustic model is responsible for outputting the Mel spectrum according to the input content audio and target timbre audio, while the vocoder converts this Mel spectrum into the corresponding audio waveform. However, this traditional voice conversion (VC) solution has the problems of low efficiency and high machine learning training cost. In particular, this traditional voice conversion (VC) solution needs to use a vocoder to convert the Mel spectrum into the target generated audio after outputting it, which greatly increases the complexity of the system and the cost of machine learning training is high.
[0075] In this regard, the embodiments of the present application provide a speech conversion model training method and device, a speech conversion method, a speech conversion model, a storage medium and an electronic device, and propose an end-to-end speech conversion solution based on adversarial training. By applying adversarial training commonly used for vocoder training (such as HiFi GAN vocoder) to acoustic model training, an end-to-end speech conversion model can be obtained, eliminating the participation of the vocoder, making the generation efficiency of the speech conversion model higher and reducing the cost of machine learning training.
[0076] Figure 1 In the illustrated embodiment, a method for training a speech conversion model is provided.
[0077] In an embodiment of the present application, the voice conversion (Voice Convention, VC) model may be an end-to-end voice conversion model, as described above. Fig.14 As shown, the voice convention (VC) model 1400 may include a content encoder 1410 , a vocal encoder 1420 , and a waveform generator 1430 .
[0078] Continue to refer Figure 1 , the voice conversion (Voice Convention, VC) model training method may include:
[0079] S110: Pre-training content encoder.
[0080] In the embodiment of the present application, the content encoder may be a separately pre-trained automatic speech recognition (ASR) model.
[0081] In some embodiments, the content encoder may be pre-trained using content training audio (not shown).
[0082] In a specific embodiment, a given loss function may be used to iteratively pre-train the content encoder. In the embodiment of the present application, step S110 may include iteratively executing the following steps until a given first pre-training convergence condition is reached: inputting the content training audio into the vocal encoder to be trained; determining the content loss from the content training audio and the content encoder output result (such as the phoneme posterior probability) according to a given loss function; and updating the parameters of the content encoder by the content loss.
[0083] In a further embodiment, multiple loss functions, such as a first content loss function and a second content loss function, may be used to iteratively train the content encoder. In a specific example, the first content loss function may be a connection temporal classification (CTC) loss function, and the second content loss function may be an attention loss function. Here, the total content loss value may be calculated by the first content loss value and the second content loss value respectively determined by the first and second content loss functions. Preferably, the first content loss value and the second content loss value may have different weights, respectively.
[0084] In an embodiment of the present application, the content encoder may be an attention-enhanced content encoder. The attention-enhanced content encoder may include: a sampling layer for sampling input audio features to obtain a downsampled feature sequence; one or more multi-head attention modules for obtaining feature associations in the input feature sequence based on attention; and a decoder for decoding the feature sequence to obtain a posterior probability of a phoneme for each frame of audio.
[0085] refer to Figure 2 and Figure 3 In the illustrated embodiment, the content encoder may be a module structure of a squeezeformer architecture.
[0086] like Figure 2 As shown, the squeezeformer includes a sampling layer, such as a depthwise separable convolution subsampling layer (DW Sep. Conv Subsampling).
[0087] Squeezeformer can also include multiple (3) multi-head attention modules, such as the Squeezeformer Block, which can be used to capture the relationship between data at different time points in the input sequence and enhance the model's ability to understand time series data. Figure 3 The various layers of the squeeze deformation module (Squeezeformer Block) are shown, which can be divided into two functional modules CF and MF. MF and CF can have the same structure, including a multi-head attention sub-module (MHAModule), a feed forward sub-module (Feed Forward Module) and a post-layer normalization layer (PostLN).
[0088] The squeezeformer may further include a decoder.
[0089] like Figure 2 As shown, the squeeze deformer (Squeezeformer) may also include an optional feed-forward structure spanning one or more (eg, 1) Squeezeformer Blocks.
[0090] In other embodiments, other attention-based content encodings may also be provided.
[0091] S120: Pre-trained vocal encoder.
[0092] In the embodiment of the present application, the voice encoder may also be referred to as a speaker encoder, which is used to extract the voice features of the speaker.
[0093] In the embodiment of the present application, the human voice feature may include a timbre feature, so that the human voice encoder may be used to extract the human voice or the speaker's timbre.
[0094] In other embodiments, the vocal features may include other vocal features in addition or in lieu thereof, which is not limited in this application. For example, in some embodiments, a voice convention (VC) model may include multiple vocal encoders, each encoder being used to extract a type of vocal feature, such as a timbre feature, which falls within the scope of this application.
[0095] In some embodiments, the vocoder may be pre-trained using speaker training audio (not shown).
[0096] In a specific embodiment, a given loss function may be used to iteratively pre-train a vocal encoder. In the embodiment of the present application, step S120 may include iteratively executing the following steps until a given second pre-training convergence condition is reached: inputting speaker training audio into the vocal encoder to be trained; determining a vocal loss value (such as a timbre loss value) from the speaker training audio and the output of the vocal encoder according to a given loss function; and updating the parameters of the vocal encoder by the vocal loss value (such as a timbre loss value).
[0097] In a further embodiment, multiple loss functions, such as a first vocal loss function and a second vocal loss function, can be used to iteratively train a vocal encoder. In a specific example, the first vocal loss function can be a Softmax loss function, and the second vocal loss function can be a Triplet loss function. Here, the total loss value can be calculated by the first loss value and the second loss value respectively determined by the first and second vocal loss functions. Preferably, the first vocal loss value and the second vocal loss value can have different weights respectively.
[0098] In an embodiment of the present application, the vocal encoder may be an attention-enhanced vocal encoder, which may include: a feature extraction layer for extracting input audio features; one or more squeeze-excited residual network blocks for enhancing given vocal features in the extracted input audio features; and an output classification layer for outputting a classification of the given vocal feature.
[0099] refer to Figure 4 and Figure 5 In the illustrated embodiment, the vocal encoder, such as the attention-enhanced vocal encoder, may be an Emphasized Channel Attention, Propagation and Aggregation in Time-Delay Neural Network (ECAPA-TDNN). Figure 4 The module structure of the ECAPA-TDNN architecture is shown.
[0100] like Figure 4 As shown, ECAPA-TDNN includes a primary feature extraction layer, which specifically includes a one-dimensional convolution layer, a ReLU layer, and a batch normalization BN layer. In this embodiment, the frame-level feature extraction layer may be responsible for extracting initial features from the input audio.
[0101] ECAPA-TDNN may also include multiple (3) squeeze-excitation residual network blocks (SE-Res2Block), which can be used to further refine and enhance vocal-related features (e.g., timbre features), which can be processed through multi-scale feature processing (squeezing and excitation) and residual mechanism. Figure 5The various layers of SE-Res2Block are shown, which may include further feature extraction layers, such as Conv1D+ReLU+BN in the 1st and 3rd layers. SE-Res2Block may also include a context expansion layer, more specifically a residual one-dimensional convolution layer with expansion, the basic structure of which is Conv1D+ReLU+BN, similar to the feature extraction layer, using expansion convolution and residual processing to capture a wider temporal context. SE-Res2Block may also include a compression excitation layer to enhance the expression of key vocal features (such as timbre features) through squeezing and excitation operations.
[0102] ECAPA-TDNN may also include a concatenation layer for concatenating outputs of multiple squeeze-excitation residual network blocks, which may include, for example, Conv1D and ReLU layers.
[0103] ECAPA-TDNN may also include an optional statistical pooling layer (which may be accompanied by a BN layer), which may, for example, convert a variable-length feature sequence into a fixed-length representation, while introducing an attention mechanism.
[0104] ECAPA-TDNN may also include a connection layer (which may have a BN layer).
[0105] ECAPA-TDNN may also include an output classification layer, such as an activation attention map Softmax (Additive Angular Margin Softmax, AAM-Softmax) layer, which is used to output the classification of a given vocal feature.
[0106] In other embodiments, other attention-based vocal encoders, in particular timbre encoders, may also be provided.
[0107] S130: adversarially training the waveform generator and the corresponding discriminator of the speech conversion model.
[0108] Combined with reference Figure 6 and Figure 7 , describes adversarial training according to an embodiment of the present application.
[0109] In the embodiment of the present application, the adversarial training includes iteratively executing several steps until a preset iteration termination condition is reached. In the embodiment of the present application, when the adversarial training is iteratively executed, the parameters of the content encoder and the vocal encoder remain fixed.
[0110] In the embodiment of the present application, adversarial training has a normal meaning known in the art. In a specific embodiment, to implement adversarial training, a discriminator may also be provided.
[0111] like Figure 6 In the illustrated embodiment, the adversarial training may include the following steps:
[0112] S131: input the training audio into the content encoder and the voice encoder respectively to output content features and voice features;
[0113] In some embodiments, the content encoder (Content Encoder) E C The output content features include the phonetic posterior probability (PPG), such as Figure 7 shown.
[0114] In some embodiments, the Speaker Encoder E s The output vocal features include the speaker timbre feature sequence Emb s ,like Figure 7 shown.
[0115] S132: Input the content features and vocal features into the waveform generator to generate training audio.
[0116] In some embodiments, before the content features and the vocal features are input into the waveform generator, the adversarial training further comprises: concatenating the content features and the vocal features, such as Figure 7 shown.
[0117] S133: The discriminator discriminates the real audio and / or the generated audio from the training audio and the training generated audio.
[0118] In a specific embodiment, the discriminator may discriminate which of the training audio and the training generated audio is the real audio (i.e., the training audio) and which is the generated audio (i.e., the training generated audio). In another embodiment, the discriminator may only discriminate the real audio. In another embodiment, the discriminator may only discriminate the generated audio. These schemes all fall within the scope of the present application.
[0119] S138: Update the parameters of the waveform generator according to the first loss value determined according to the first correlation of the discrimination result.
[0120] In some embodiments, the waveform generator may be the same type of G waveform generator used in the HiFi-GAN model.
[0121] S139: Update the parameters of the discriminator according to a second loss value determined according to a second correlation of the discrimination result.
[0122] In this embodiment, the first correlation and the second correlation of steps S138 and S139 are opposite. Thus, the updating of the waveform generator parameters of the speech conversion model is determined based on the degree of incorrect discrimination of the discriminator; and the updating of the discriminator parameters is determined based on the degree of correct discrimination of the discriminator. Here, if the discrimination result is that the discriminator correctly discriminates the real video as correct, the first correlation is negative correlation and the second correlation is positive correlation.
[0123] Specifically, the training audio x and the training generated audio x' (also the output G(E of the waveform generator) can be used to generate the training audio x and the training generated audio x') C (x),E S (x)) is characterized by using the loss functions for the speech conversion model (preferably the waveform generator) and the discriminator, namely the first loss function (such as the generator loss function L G ) and the second loss function (such as the discriminator loss function L D ), respectively determine a first loss value for iteratively updating the speech conversion model (preferably a waveform generator) and a second loss value for iteratively updating the discriminator.
[0124] In a preferred embodiment, Figure 7 As shown, the generator loss function L G And the discriminator loss function L D Can be inverse.
[0125] In a specific embodiment, the generator loss function L G and the discriminator loss function L D Satisfy the following formula:
[0126]
[0127] In a further embodiment, additional iteration constraints may be added to expect that the text content of the output text of the speech conversion model is more consistent with the target text.
[0128] like Figure 8 Another embodiment of adversarial training is shown, which may include the following steps:
[0129] S131: Input the training audio into the content encoder and the voice encoder respectively to output content features and voice features.
[0130] S132: Input the content features and vocal features into the waveform generator to generate training audio.
[0131] S133: The discriminator discriminates the real audio and / or the generated audio from the training audio and the training generated audio;
[0132] Figure 8Steps S131-S1333 of the illustrated embodiment may refer to Figure 6 The corresponding steps of the illustrated embodiment are not described in detail here.
[0133] S134: Input the training generated audio into the content encoder to obtain generated content features.
[0134] like Figure 7 As shown, in this preferred embodiment, the training generates audio x' (which can also be the output G (E C (x), E S (x)) is used to represent that in addition to the discriminator, it will also be input to the content encoder (ContentEncoder) E C In order to generate the generated content feature PPG corresponding to the training generated audio, it can be used as E C (G(E C (x), E S (x))) indicates.
[0135] S135: Determine a third loss value according to the content feature and the generated content feature.
[0136] Accordingly, a given third loss function (content loss function) can be used to determine a corresponding third loss value based on the content features and the generated content features.
[0137] In some embodiments, the third loss function (content loss function) may be a mean square error (MSE) loss function.
[0138] For example, the third loss function (content loss function L C ) can be expressed by the following formula:
[0139] L C =MSE(E C (x), E C (G(E C (x), E S (x))))
[0140] However, the present application is not limited to this, and the third loss function (content loss function) may also be, for example, one or more of a root mean square error (RMSE) loss function, a mean absolute error (MAE) loss function, and a mean absolute percentage error (MAPE) loss function.
[0141] S138': Update the parameters of the waveform generator according to the first loss value and the third loss value.
[0142] Figure 8 The determination of the first loss value described in the embodiment can refer to Figure 6 The illustrated embodiments are not described in detail here.
[0143] and Figure 6 The difference of step S138 in the embodiment shown is that Figure 8 In the illustrated embodiment, the parameters of the waveform generator in step S138 ′ are updated according to the common definition of the first loss value and the third loss value.
[0144] For example, the total loss value LT determined by the first loss value and the third loss value can be determined according to the following formula:
[0145] L T =αL D +βL C
[0146] Among them, α and β are weight values.
[0147] However, other schemes can be thought of in which the first loss value and the third loss value jointly constrain the parameter update of the waveform generator. For example, in some embodiments, the third loss value may have different weights according to different discrimination results of step S133. In other embodiments, the third loss value may be considered based on the condition of the discrimination result, so that, for example, when the discriminator correctly discriminates that the training audio x is real audio, the third loss value may not be considered.
[0148] S139: Update the parameters of the discriminator according to a second loss value determined according to a second correlation of the discrimination result.
[0149] Figure 8 Step S139 of the illustrated embodiment can refer to Figure 6 The corresponding steps of the illustrated embodiment are not described in detail here.
[0150] In other embodiments, a vocal encoder may be used to add additional iteration constraints in order to expect that the vocal features (such as timbre) of the output audio of the speech conversion model will be more consistent with the vocal features of the target input audio.
[0151] like Fig. 9 Another embodiment of adversarial training is shown, which may include the following steps:
[0152] S131: Input the training audio into the content encoder and the voice encoder respectively to output content features and voice features.
[0153] S132: Input the content features and vocal features into the waveform generator to generate training audio.
[0154] S133: The discriminator discriminates the real audio and / or the generated audio from the training audio and the training generated audio.
[0155] Fig. 9 Steps S131-S133 of the illustrated embodiment may refer to Figure 6 and Figure 8 The corresponding steps of the illustrated embodiment are not described in detail here.
[0156] S136: Input the training generated audio into the vocal encoder to obtain generated vocal features.
[0157] In the preferred embodiment not shown, the training generated audio x' (can also be the output G (E of the waveform generator) C (x), E S (x)) is used to represent that in addition to the discriminator, it will also be input to the speaker encoder E s In order to generate the generated vocal features corresponding to the training generated audio, such as generating the timbre coding sequence Emb s , can be E S (G(E C (x), E S (x))) indicates.
[0158] S137: Determine a fourth loss value based on the vocal feature and the generated vocal feature.
[0159] Accordingly, a given fourth loss function (voice loss function) can be used to determine a corresponding fourth loss value by the vocal features and the generated vocal features.
[0160] In some embodiments, the fourth loss function (voice loss function) may be a mean square error (MSE) loss function.
[0161] For example, the fourth loss function (the vocal loss function L S ) can be expressed by the following formula:
[0162] L S =MSE(E s (x), E s (G(E C (x), E S (x))))
[0163] However, the present application is not limited to this. The fourth loss function (voice loss function) may also be, for example, one or more of a root mean square error (RMSE) loss function, a mean absolute error (MAE) loss function, and a mean absolute percentage error (MAPE) loss function.
[0164] S138”: Update the parameters of the waveform generator according to the first loss value and the fourth loss value.
[0165] Fig. 9 The determination of the first loss value described in the embodiment can refer to Figure 6 and Figure 8 The illustrated embodiments are not described in detail here.
[0166] and Figure 6 Step S138 of the embodiment shown and Figure 8 The step S138′ of the embodiment shown is different in that Fig. 9 In step S138" of the illustrated embodiment, the parameters of the waveform generator will be updated according to the common definition of the first loss value and the fourth loss value.
[0167] For example, the total loss value L' determined by the first loss value and the fourth loss value T It can be determined according to the following formula:
[0168] L′ T =αL D +βL s
[0169] Among them, α and β are weight values.
[0170] However, other schemes can be thought of in which the first loss value and the fourth loss value jointly constrain the parameter update of the waveform generator. For example, in some embodiments, the fourth loss value may have different weights according to different discrimination results of step S133. In other embodiments, the fourth loss value may be considered based on the condition of the discrimination result, so that, for example, when the discriminator correctly discriminates that the training audio x is real audio, the fourth loss value may not be considered.
[0171] S139: Update the parameters of the discriminator according to a second loss value determined according to a second correlation of the discrimination result.
[0172] Fig. 9 Step S139 of the illustrated embodiment can refer to Figure 6 and Figure 8 The corresponding steps of the illustrated embodiment are not described in detail here.
[0173] In other embodiments, the content encoder and the vocal encoder may be used together to impose additional iteration constraints.
[0174] like Fig.10 Another embodiment of adversarial training is shown, which may include the following steps:
[0175] S131: Input the training audio into the content encoder and the voice encoder respectively to output content features and voice features.
[0176] S132: Input the content features and vocal features into the waveform generator to generate training audio.
[0177] S133: The discriminator discriminates the real audio and / or the generated audio from the training audio and the training generated audio.
[0178] S134: Input the training generated audio into the content encoder to obtain generated content features.
[0179] S135: Determine a third loss value according to the content feature and the generated content feature.
[0180] S136: Input the training generated audio into the vocal encoder to obtain generated vocal features.
[0181] S137: Determine a fourth loss value based on the vocal feature and the generated vocal feature.
[0182] Fig.10 Steps S131-S133 of the illustrated embodiment may refer to Figure 6 , Figure 8 and Fig. 9 The corresponding steps of the illustrated embodiment are not described in detail here.
[0183] Fig.10 Steps S134-S135 of the illustrated embodiment may refer to Figure 8 The corresponding steps of the illustrated embodiment are not described in detail here.
[0184] Fig.10 Steps S136-S137 of the illustrated embodiment may refer to Fig. 9 The corresponding steps of the illustrated embodiment are not described in detail here.
[0185] S138'": Update the parameters of the waveform generator according to the first loss value, the third loss value and the fourth loss value.
[0186] exist Fig.10 In the embodiment shown, Figure 8 and Fig. 9The embodiment shown is different in that, in step S138'", the parameters of the waveform generator are updated by being constrained by the first loss value, the third loss value and the fourth loss value.
[0187] Fig.10 The determination of the first loss value described in the embodiment can refer to Figure 6 , Figure 8 and Fig. 9 The illustrated embodiments are not described in detail here.
[0188] Fig.10 The determination of the third loss value described in the embodiment can refer to Figure 8 The illustrated embodiments are not described in detail here.
[0189] Fig.10 The determination of the fourth loss value described in the embodiment can refer to Fig. 9 The illustrated embodiments are not described in detail here.
[0190] For example, the total loss value L" determined by the first loss value, the third loss value, and the fourth loss value T It can be determined according to the following formula:
[0191] L′ T =αL D +βL C +γL S
[0192] Among them, α, β, and γ are weight values.
[0193] However, other schemes can be thought of in which the first loss value, the third loss value, and the fourth loss value jointly constrain the parameter update of the waveform generator. For example, in some embodiments, the third loss value and the fourth loss value may have different weights according to different discrimination results of step S133. In other embodiments, the third loss value and the fourth loss value may be considered based on the conditions of the discrimination result, so that, for example, when the discriminator correctly discriminates that the training audio x is real audio, the third loss value and the fourth loss value may not be considered.
[0194] S139: Update the parameters of the discriminator according to a second loss value determined according to a second correlation of the discrimination result.
[0195] Fig.10 Step S139 of the illustrated embodiment can refer to Figure 6 , Figure 8 and Fig. 9 The corresponding steps of the illustrated embodiment are not described in detail here.
[0196] In another embodiment of the present application, a speech conversion method may be provided, which may use a speech conversion model obtained by executing the training method described in the embodiment of the present application.
[0197] like Fig.11 As shown, a speech conversion method may include:
[0198] S1110: Input the target content audio into the content encoder of the speech conversion model to output target content features.
[0199] S1120: Input the target vocal audio into the vocal encoder of the speech conversion model to output target vocal features.
[0200] S1130: Input the target content features and target vocal features into the waveform generator of the speech conversion model to generate target generated audio.
[0201] Before step S1130, the speech conversion method further includes: concatenating the target content feature and the target vocal feature.
[0202] In the present embodiment, when the voice conversion method is executed, the discriminator will be discarded.
[0203] like Fig.12 As shown, the target vocal audio X1, for example, has the vocal characteristics expected to be possessed by the generated speech, such as timbre, and can be input into a trained vocal encoder. The target content audio X2, for example, has the content expected to be possessed by the generated speech, and can be input into a trained content encoder. The outputs of the vocal encoder and the content encoder are spliced and input into the G waveform generator to obtain the target generated audio, which has the content or similar content of X2 on the one hand, and the vocal characteristics, such as timbre or similar vocal characteristics, such as similar timbre, of X1 on the other hand. Moreover, the speech conversion method will be end-to-end without the need for other pre-processing and / or post-processing.
[0204] refer to Fig.13, also shows a speech conversion model training device 1300 including a content encoder, a vocal encoder and a waveform generator according to an embodiment of the present application, which may include a first pre-training unit 1310 for pre-training the content encoder; a second pre-training unit 1320 for pre-training the vocal encoder; and an adversarial training unit 1330 for adversarially training the waveform generator and the corresponding discriminator of the speech conversion model. The adversarial training unit 1330 is configured to iteratively perform the following steps until a preset iteration termination condition is reached: inputting the training audio into the content encoder and the vocal encoder respectively to output content features and vocal features; inputting the content features and vocal features into the waveform generator to generate training generated audio, and the discriminator discriminates the real audio and / or generated audio from the training audio and the training generated audio; updating the parameters of the waveform generator by a first loss value determined according to a first correlation of the discrimination result; and updating the parameters of the discriminator by a second loss value determined according to a second correlation of the discrimination result.
[0205] Optionally, the speech conversion model training device 1300 may further include a concatenation unit (not shown) for concatenating content features and vocal features.
[0206] like Fig.14 14, and a speech conversion model 1400 of an embodiment of the present application is also shown, which may include a content encoder 1410, a vocal encoder 1420, and a waveform generator 1430. It is understood that a training device 1300 of an embodiment of the present specification may also execute Figures 1 to 10 The features of the training method, and the features of the devices, models, systems, components, etc. of the embodiments of the present application can refer to the features of the methods, steps, etc. of the embodiments of the present application, and the device, model, and system embodiments can be combined with the features of the method embodiments to obtain new embodiments, and vice versa, and will not be repeated here.
[0207] In an embodiment of the present application, an electronic device is provided, comprising: a processor and a memory storing a computer program, wherein the processor is configured to execute the speech synthesis model training method of any embodiment and the speech synthesis method of any embodiment when running the computer program.
[0208] Fig.15 A schematic diagram of an electronic device 1500 that can implement a method or implement an embodiment of the present application is shown, and in some embodiments, more or fewer electronic devices than shown may be included. In some embodiments, a single or multiple electronic devices may be used for implementation. In some embodiments, a cloud or distributed electronic device may be used for implementation.
[0209] like Fig.15As shown, the electronic device 1500 includes a processor 1501, which can perform various appropriate operations and processes according to the programs and / or data stored in the read-only memory (ROM) 1502 or the programs and / or data loaded from the storage part 1508 to the random access memory (RAM) 1503. Processor 1501 can be a multi-core processor or can include multiple processors. In some embodiments, processor 1501 can include a general main processor and one or more special coprocessors, such as a central processing unit (CPU), a graphics processing unit (GPU), a neural network processor (NPU), a digital signal processor (DSP), etc. In RAM 1503, various programs and data required for the operation of electronic device 1500 are also stored. Processor 1501, ROM 1502 and RAM 1503 are connected to each other via bus 1504. Input / output (I / O) interface 1505 is also connected to bus 1504.
[0210] The processor and the memory are used together to execute the program stored in the memory. When the program is executed by the computer, the methods, steps or functions described in the above embodiments can be implemented.
[0211] The following components are connected to the I / O interface 1505: an input section 1506 including a keyboard, a mouse, a touch screen, etc.; an output section 1507 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1508 including a hard disk, etc.; and a communication section 1509 including a network interface card such as a LAN card, a modem, etc. The communication section 1509 performs communication processing via a network such as the Internet. A drive 1510 is also connected to the I / O interface 1505 as needed. A removable medium 1511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1510 as needed, so that a computer program read therefrom is installed into the storage section 1508 as needed. Fig.15 Only some components are schematically shown in the figure, which does not mean that the computer system 1500 only includes Fig.15 Components shown.
[0212] The systems, devices, modules or units described in the above embodiments may be implemented by a computer or its associated components. The computer may be, for example, a mobile terminal, a smart phone, a personal computer, a laptop computer, a vehicle-mounted human-computer interaction device, a personal digital assistant, a media player, a navigation device, a game console, a tablet computer, a wearable device, a smart TV, an Internet of Things system, a smart home, an industrial computer, a server or a combination thereof.
[0213] Although not shown, in an embodiment of the present application, a storage medium is provided, the storage medium storing a computer program, the computer program being configured to execute the speech conversion model training method of any embodiment or the speech conversion method of any embodiment when executed.
[0214] The storage media in the embodiments of the present application include permanent and non-permanent, removable and non-removable items that can be used to store information by any method or technology. Examples of storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission medium that can be used to store information that can be accessed by a computing device.
[0215] The methods, programs, systems, devices, etc. of the embodiments of the present application may be executed or implemented in a single or multiple networked computers, or may be practiced in a distributed computing environment. In the embodiments of the present specification, in these distributed computing environments, tasks may be performed by remote processing devices connected via a communication network.
[0216] Those skilled in the art should understand that the embodiments of the present specification may be provided as methods, systems or computer program products. Therefore, those skilled in the art may imagine that the functional modules / units or controllers and related method steps described in the above embodiments may be implemented in software, hardware or a combination of software / hardware.
[0217] Unless explicitly stated, the actions or steps of the methods, programs, and embodiments of the present application do not have to be performed in a specific order and can still achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
[0218] In this article, multiple embodiments of the present application are described, but for the sake of simplicity, the description of each embodiment is not exhaustive, and the same or similar features or parts between the various embodiments may be omitted. In this article, "one embodiment", "some embodiments", "example", "specific example", or "some examples" are intended to be applicable to at least one embodiment or example according to the present application, rather than all embodiments. The above terms do not necessarily mean to refer to the same embodiment or example. In the absence of mutual contradiction, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples.
[0219] The exemplary systems and methods of the present application have been specifically shown and described with reference to the above embodiments, which are merely examples of the best modes for implementing the present systems and methods. It will be appreciated by those skilled in the art that various changes may be made to the embodiments of the systems and methods described herein when implementing the present systems and / or methods without departing from the spirit and scope of the present application as defined in the appended claims.
Claims
1. A speech conversion model training method, characterized in that: The speech conversion model includes a content encoder, a vocal encoder and a waveform generator; The training method comprises: pre-training the content encoder; pre-training the vocal encoder; as well as The waveform generator and the corresponding discriminator of the speech conversion model are adversarially trained, wherein the adversarial training comprises iteratively executing the following steps until a preset iteration termination condition is reached: Input the training audio into the content encoder and the vocal encoder respectively to output content features and vocal features; Input the content features and vocal features into the waveform generator to generate training audio, The discriminator discriminates the real audio and / or the generated audio from the training audio and the training generated audio; updating the parameters of the waveform generator according to a first loss value determined according to a first correlation of the discrimination result; The parameters of the discriminator are updated by a second loss value determined according to a second correlation of the discrimination result.
2. The method according to claim 1, characterized in that When the adversarial training is iteratively performed, the parameters of the content encoder and the vocal encoder remain fixed.
3. The method according to claim 1 or 2, characterized in that: The adversarial training also includes: Inputting the training generated audio into the content encoder to obtain generated content features; determining a third loss value according to the content feature and the generated content feature; Wherein, updating the parameters of the waveform generator according to the first loss value determined according to the first correlation of the discrimination result includes: Parameters of the waveform generator are updated by the first loss value and the third loss value.
4. The method according to claim 1 or 2, characterized in that: The adversarial training also includes: Inputting the training generated audio into the vocal encoder to obtain generated vocal features; Determine a fourth loss value according to the vocal feature and the generated vocal feature; Wherein, updating the parameters of the waveform generator according to the first loss value determined according to the first correlation of the discrimination result includes: Parameters of the waveform generator are updated according to the first loss value and the fourth loss value.
5. The method according to claim 1 or 2, characterized in that: The adversarial training also includes: Inputting the training generated audio into the content encoder to obtain generated content features; determining a third loss value according to the content feature and the generated content feature; Inputting the training generated audio into the vocal encoder to obtain generated vocal features; Determine a fourth loss value according to the vocal feature and the generated vocal feature; Wherein, updating the parameters of the waveform generator according to the first loss value determined according to the first correlation of the discrimination result includes: Parameters of the waveform generator are updated by the first loss value, the third loss value, and the fourth loss value.
6. The method according to claim 1 or 2, characterized in that: The content encoder is an attention-enhanced content encoder, comprising: A sampling layer, used to sample the input audio features to obtain a downsampled feature sequence; One or more multi-head attention modules for acquiring feature associations in an input feature sequence based on attention; and A decoder is used to decode the feature sequence to obtain the posterior probability of the phoneme of each frame of audio.
7. The method according to claim 1 or 2, characterized in that: The human voice encoder is an attention-enhanced human voice encoder, comprising: Feature extraction layer, used to extract input audio features; One or more squeeze-excited residual network blocks for enhancing given vocal features in the extracted input audio features; and The output classification layer is used to output the classification of given vocal features.
8. The method according to claim 1 or 2, characterized in that: The vocal feature includes a speaker's timbre feature sequence, and / or the content feature includes a phoneme posterior probability.
9. The method according to claim 1 or 2, characterized in that: Before inputting the content features and the vocal features into the waveform generator, the adversarial training further includes: concatenating the content features and the vocal features.
10. A voice conversion method, characterized in that: include: Inputting the target content audio into a content encoder of a speech conversion model to output target content features, wherein the speech conversion model is obtained by executing the method according to any one of claims 1 to 9; Inputting the target human voice audio into the human voice encoder of the speech conversion model to output the target human voice features; as well as The target content features and target vocal features are input into the waveform generator of the speech conversion model to generate target generated audio.
11. The method according to claim 10, characterized in that Before inputting the target content feature and the target vocal feature into the waveform generator of the speech conversion model, the method further includes: The target content feature and the target vocal feature are concatenated.
12. A speech conversion model, characterized in that: It comprises a content encoder, a vocal encoder and a waveform generator, wherein the speech conversion model is trained by the training method according to any one of claims 1 to 9.
13. An electronic device, characterized in that: The invention comprises a processor and a memory storing a computer program, wherein the processor is configured to execute the method according to any one of claims 1 to 11 when running the computer program.
14. A storage medium, characterized in that: The storage medium stores a computer program, and the computer program is configured to execute the method according to any one of claims 1 to 11 when executed.
