A speech conversion method for personalized speech generation

Abstract

The present application uses a phoneme-based information bottleneck to characterize speaker style and control the speed of converted speech, consisting of a content encoder, a speaker encoder, a phoneme duration converter, a decoder and a vocoder. The content information is separated from the source speech through the information bottleneck between the encoder and the decoder, and is input into the decoder together with the target speaker embedding, and finally input into the vocoder and generate the converted speech. By introducing the duration converter, the present application uses a designed phoneme-level information bottleneck to separate the speaker content information and the speaker style information, which can be applied to zero-shot learning and is also applicable to speech outside the training data set.

Description

Technical Field

[0001] This invention relates to the field of speech conversion, and more specifically to a speech conversion method for personalized speech generation. Background Technology

[0002] With the increasing adoption of AI products, the need for personalized voice is growing. Speech conversion is a powerful technique for generating personalized speech. It transforms the speech of a source speaker into that of a target speaker without altering the content, making the converted speech sound identical to the target speaker's voice. Traditional speech conversion methods use statistical models to explicitly represent the conversion function, converting the acoustic features of the source speaker into those of the target speaker. Recent research has replaced traditional methods with deep neural networks, such as feedforward neural networks and recurrent neural networks. The introduction of vocoders like WaveNet has significantly improved the quality of generated speech. However, most conversion methods rely on parallel corpora for training, requiring both the source and target speakers to provide audio data containing the same utterances. Non-parallel speech conversion techniques, unlike parallel methods, do not have these limitations of parallel data and time alignment, making data collection simpler and less costly. Summary of the Invention

[0003] This invention utilizes a phoneme-based information bottleneck to characterize speaker style and control the speed of speech conversion. It consists of a content encoder, a speaker encoder, a phoneme duration converter, a decoder, and a vocoder. Content information is extracted from the original speech through the information bottleneck between the encoder and decoder, and this content, along with the target speaker embedding, is input into the decoder. Finally, it is input into the vocoder to generate the converted speech. The content encoder, composed of a Conformer module, converts the source speaker's Mel spectrogram into a hidden representation sequence. The phoneme duration converter, consisting of a downsampler, an upsampler, a phoneme predictor, and a duration predictor, controls the duration of speech phonemes and simultaneously converts the speaker's style. The decoder, also composed of a Conformer module, converts the target speaker's hidden sequence representation into a Mel spectrogram. To further refine the initially estimated and reconstructed Mel spectrogram, PostNet is used at the decoder's end to improve the quality of the generated Mel spectrogram. The loss function of the entire conversion system comprises self-reconstruction loss, phoneme prediction loss, duration prediction loss, periodic cycle consistency loss, and speaker classification loss.

[0004] This invention proposes a speech conversion method for personalized speech generation, comprising the following steps:

[0005] Step 1: Train the conversion model, obtain the source speaker's speech database, and extract the source speaker's Mel spectrogram sequences X from two different segments.u =[x u ,x u+1 ,…,x u+N ] and X v =[x v ,x v+1 ,…,x v+N ], which are used as speech features for training; where u and v are the starting numbers of the Mel spectrogram sequence, and N is the length of the extracted Mel spectrogram sequence;

[0006] Step 2: The extracted Mel spectrum sequence is processed by the content encoder E. c Encode the Mel spectrum into a Mel spectrum hidden sequence, represented as:

[0007] C u =E c (X u ) ⑼

[0008] C v =E c (X v ) 10

[0009] Step 3: Simultaneously input the source speaker's speech from these two different segments into the speaker encoder E. s Get speaker embedding S u and S v , is represented as:

[0010] S u =E s (X u ) ⑾

[0011] S v =E s (X v ) ⑿

[0012] Step 4: Embed the content of the corresponding segment and the speaker in C. u S u With C v S v Input into the phoneme duration converter, and convert C u The phoneme duration in the S v The duration of the phoneme, similarly for C v The phoneme duration in the S u The phoneme duration is calculated, and the output is the C after phoneme duration conversion. u ′ and C v ′;

[0013] Step 5, set the corresponding C u ′、S u C v ′、Sv The input decoder utilizes the overall loss function L total Training is performed where, to refine the reconstructed Mel spectra after the initial estimation, PostNet is used at the end of the decoder to improve the quality of the generated Mel spectra. The overall loss function is shown below:

[0014] L total =L recon +λ p L pho +λ d L d +L cycle +L id 13

[0015] Complete the training;

[0016] Step 6, actual conversion: obtain the original speaker's speech Mel spectrogram X and the target speaker's speech Mel spectrogram Y. Input the original speaker's speech Mel spectrogram X into the content encoder to obtain the content embedding C. Input the target speaker's speech Mel spectrogram Y into the speaker encoder to obtain the speaker embedding S.

[0017] Step 7: Simultaneously input the content embedding C and the speaker embedding S into the phoneme duration converter to obtain the content embedding C′ after phoneme duration conversion;

[0018] Step 8: Simultaneously input the converted phoneme duration content into C′ and the speaker embedding S into the decoder to obtain the converted Mel spectrogram Y′;

[0019] Step 9: Input the converted Mel spectrogram Y′ obtained in step 8 into the vocoder to synthesize the speech waveform and obtain high-quality speech similar to the target speaker.

[0020] Preferably, step 2 includes the following sub-steps:

[0021] Sub-step A1: Hide the Mel spectrum sequence C mel Input phoneme predictor;

[0022] Sub-step A2: Use the phoneme predictor to obtain the Mel spectrogram hidden sequence C. mel Corresponding phoneme tag sequence

[0023] Sub-step A3, hide the Mel spectrum sequence C mel Corresponding phoneme tag sequence Simultaneously, the input is downsampled in the downsampler to remove the speaker's phoneme duration information, resulting in the phoneme hidden sequence C. pho ;

[0024] Sub-step A4: Input the target speaker information obtained by the speaker encoder into the duration predictor to obtain the phoneme duration sequence of the target speaker.

[0025] Sub-step A5, hide the phoneme sequence C pho and the phoneme duration sequence of the target speaker Simultaneously, the input upsampler is used to upsample and add the phoneme duration information of the target speaker, resulting in the duration-converted Mel spectrogram hidden sequence. As a preferred option, the phoneme predictor P pho Used to predict C mel phoneme tag sequence The phoneme tag sequence obtained by optimization prediction With the real phoneme label sequence P = [p1, p2, ..., p N The phoneme predictor is trained using the phoneme classification loss between [ ], which is expressed as:

[0026]

[0027] Where N is the length of the Mel spectrum of the hidden sequence, P pho This represents a phoneme predictor;

[0028] Given a predicted phoneme hiding sequence Then, the downsampler DS is used to sample C. mel Perform downsampling; the downsampling process is represented as follows:

[0029]

[0030] Where C mel Hidden sequences for Mel spectra, Given a sequence of predicted phoneme labels, DS is a downsampler. Let M be the hidden sequence of the i-th phoneme, and M be the length of the hidden sequence of the phoneme.

[0031] In order to hide the phoneme sequence C pho Expand back to hidden sequence C mel The duration predictor P was used. d and a phoneme-level duration-based upsampler US;

[0032] In order to predict C using information from the target speaker pho The corresponding phoneme duration sequence, C after downsampling pho Both the speaker embedding S and the duration predictor P are input into the duration predictor. d middle;

[0033] P d The prediction corresponds to C phoPhoneme duration sequence By embedding the source speaker into P d To predict source speaker information The duration sequence of the actual phonemes is D = [d1, d2, ..., d]. M The duration predictor is trained using the L1 loss between [ ], and the loss function is expressed as:

[0034]

[0035] Where M is the length of the phoneme hidden sequence, P d For duration predictor, Let d be the hidden sequence of the i-th phoneme. i The duration of the i-th phoneme;

[0036] Upsampler US will C pho Expanded to be based on Extended Mel spectrum hidden sequence The upsampling process is represented as:

[0037]

[0038] Where K is the length of the Mel spectrogram sequence, which is equal to N during the training phase, α is a hyperparameter controlling the extension time of each phoneme, and C... pho For phoneme hiding sequences, US is the upsampler. For the hidden sequence of the i-th Mel spectrum, It is a sequence of phoneme durations.

[0039] Preferably, the human encoder E described in step 3 s The encoding process includes the following sub-steps:

[0040] Sub-step B1: Input the target speaker Mel spectrogram X into the speaker encoder. The Mel spectrogram X is processed through three layers of LSTM to obtain X1.

[0041] Sub-step B2 extracts the last layer output of sub-step B1, denoted as X2;

[0042] Sub-step B3, X2 passes through the Linear layer to obtain the speaker embedding representation S.

[0043] In sub-step B4, in order to extract the same speaker embedding information from the same speaker, 10 speech samples from the same speaker are input into the speaker encoder to obtain 10 speaker embeddings, and the average value of the 10 speaker embeddings is taken as the speaker embedding representation of the current speaker.

[0044] Preferably, the content encoder's specific processing includes the following sub-steps:

[0045] Sub-step C1: Input the original speaker Mel spectrogram X into the content encoder, and pass it through the feedforward module, which includes the LayerNorm layer, the Linear layer, the Swish activation layer and the Dropout layer, to obtain X1 = X + FFN(X), where FFN represents the feedforward module;

[0046] Sub-step C2, after passing through the multi-head self-attention module, including the LayerNorm layer, the multi-head self-attention layer, and the Dropout layer, yields X2 = X1 + MHA(X1), where MHA represents the multi-head self-attention module.

[0047] In sub-step C3, X2 passes through a convolutional module to obtain X3 = X2 + Conv(X2), where Conv represents the convolutional module, which includes LayerNorm, 1D convolutional layers, gated linear unit layers, InstanceNorm, and Dropout layers.

[0048] Sub-step C4 involves connecting the speaker's embedded S and simultaneously inputting it into the feedforward module to obtain... Indicates a Concat connection

[0049] Sub-step C5, after passing through the LayerNorm layer, yields the content embedding C.

[0050] Preferably, the decoder's specific processing includes the following sub-steps:

[0051] Sub-step D1: Input the content information C after duration conversion and the target speaker information S into the content encoder. The content information passes through the C feedforward module, which includes the LayerNorm layer, Linear layer, Swish activation layer and Dropout layer, to obtain C1 = C + FFN(C), where FFN represents the feedforward module.

[0052] In sub-step D2, C1 passes through the multi-head self-attention module, which includes the LayerNorm layer, the multi-head self-attention layer, and the Dropout layer, to obtain C2 = C1 + MHA(C1), where MHA represents the multi-head self-attention module;

[0053] In sub-step D3, C2 passes through a convolutional module to obtain C3 = C2 + Conv(C2), where Conv represents the convolutional module, which includes LayerNorm, 1D convolutional layer, gated linear unit layer, InstanceNorm layer, and Dropout layer.

[0054] Sub-step D4 involves connecting the speaker's embedded S and simultaneously inputting it into the feedforward module to obtain... in Indicates a Concat connection;

[0055] Sub-steps D5 and C4 pass through LayerNorm and PostNet to obtain the transformed Mel spectrum. PN stands for PostNET network, which includes 1d convolutional layers and BatchNorm layers.

[0056] Preferably, step 6 further includes style transfer, which includes the following sub-steps:

[0057] Sub-step E1, embedding S in the speaker before conversion u and S v Add a projection vector at the very beginning to represent the speaker's identity information;

[0058] Sub-step E2 involves inputting the converted speaker Mel spectrogram sequence into the speaker encoder to obtain the converted speaker embedding representation, which is S. u ′=E s (X u ′) and S v ′=E s (X v ′);

[0059] Sub-step E3, with periodic consistency loss L cycle Embedding S in the transformed speaker u ′ and S v Speaker embedding with the original speaker u and S v Calculate the speaker's losses;

[0060] Sub-step E4: Extract the transformed speaker embedding S u ′ and S v The projection vector of ′, with identity loss L id Embedding S in the transformed speaker u ′ and S v Speaker embedding with the original speaker u and S v Perform an identity loss calculation.

[0061] The essential features of this invention are as follows: (1) This invention introduces a phoneme duration converter and uses a well-designed phoneme-level information bottleneck to separate speaker content information and speaker style information. It can be applied to zero-time learning and is also applicable to speech outside the training dataset. (2) In the training process, this invention uses a new self-reconstruction loss to train the overall model, and at the same time adds periodic consistency loss and identity loss to further separate speaker content and speaker information, so that the two can be separated more effectively and the target can be better converted. (3) This invention uses a new content encoder and decoder for the encoding and decoding process, which improves the conversion performance. It not only reduces the computation time, but also the conversion method does not require parallel speech data, which reduces the data collection cost and effectively saves resources and time costs. Attached Figure Description

[0062] Figure 1 Flowchart of the system training phase of this invention;

[0063] Figure 2 Flowchart of the system conversion stage of this invention;

[0064] Figure 3 The time converter structure diagram of the present invention;

[0065] Figure 4 Structural diagrams of the phoneme predictor and duration predictor of the present invention;

[0066] Figure 5 Structural diagram of the speaker encoder of the present invention;

[0067] Figure 6 The content encoder structure diagram of the present invention;

[0068] Figure 7 The decoder structure diagram of this invention;

[0069] Figure 8 Detailed training process flowchart. Detailed Implementation

[0070] The technical solution of the present invention will be further described in detail below through specific embodiments and in conjunction with the accompanying drawings.

[0071] Example 1

[0072] The flowchart of the speech conversion system of the present invention during the training phase is as follows: Figure 1 As shown, the flowchart for the transformation phase is as follows: Figure 2 As shown, the contents of each major module of the system will be described in detail below.

[0073] Phoneme Duration Converter:

[0074] In some novel speech conversion models, a suitable information bottleneck is formed by adjusting the downsampling factor to separate speaker content information from speaker style information. However, due to its high dependence on the information bottleneck, a suitable downsampling factor must be found through multiple experiments to achieve high-performance conversion. A large downsampling factor degrades audio quality, while a small downsampling factor degrades conversion performance. This invention uses frame-level phoneme information for transmission and conversion, proposing a frame-level phoneme downsampling method to separate content and speaker information from the Mel spectrogram, improving audio quality and speech conversion performance even without finding a suitable downsampling factor.

[0075] To enable speech conversion with controllable phoneme duration, this invention incorporates a phoneme duration converter. It consists of a frame-level downsampler, a phoneme-level upsampler, a phoneme predictor, and a duration predictor, as shown in the specific structure below. Figure 3 As shown in the figure. The structures of the phoneme predictor and the duration predictor are as follows: Figure 4 As shown.

[0076] The extraction and use of frame-level information bottlenecks can be divided into two processes: downsampling and upsampling. For the downsampling process, let C... mel =[h1,h2,…,h N [This refers to the hidden sequence in the Mel spectrum.] Let N be the phoneme hiding sequence, where N and M are the lengths of the Mel spectrogram and the phoneme hiding sequence, respectively.

[0077] To perform frame-level downsampling, this invention uses a phoneme predictor P. pho Used to predict C mel phoneme tag sequence And by optimizing the predicted phoneme tag sequence With the real phoneme label sequence P = [p1, p2, ..., p N The phoneme predictor is trained using the phoneme classification loss between [ ], which is expressed as:

[0078]

[0079] Where N is the length of the Mel spectrum of the hidden sequence, P pho This represents a phoneme predictor. Given a hidden sequence of phonemes to predict... Then, the downsampler DS is used to sample C. mel Perform downsampling; the downsampling process is represented as follows:

[0080]

[0081] Where C mel Hidden sequences for Mel spectra, Given a sequence of predicted phoneme labels, DS is a downsampler. Let M be the hidden sequence of the i-th phoneme, and M be the length of the hidden sequence of the phoneme.

[0082] In order to hide the phoneme sequence C pho Expand back to hidden sequence C mel The duration predictor P was used. d And a phoneme-level duration-based upsampler US. The number of spectrogram frames mapped to each phoneme is called the phoneme duration. This is to predict C using information from the target speaker. pho The corresponding phoneme duration sequence, C after downsampling pho Both the speaker embedding S and the duration predictor P are input into the duration predictor. d Then, taking into account the information embedded by the target speaker, P d The prediction corresponds to C pho Phoneme duration sequence By embedding the source speaker into P d To predict source speaker information The duration sequence of the actual phonemes is D = [d1, d2, ..., d]. M The duration predictor is trained using the L1 loss between [ ], and the loss function is expressed as:

[0083]

[0084] Where M is the length of the phoneme hidden sequence, P d For duration predictor, Let d be the hidden sequence of the i-th phoneme. i Let be the duration of the i-th phoneme.

[0085] Once the predicted phoneme duration sequence is determined Upsampler US will C pho Expanded to be based on Extended Mel spectrum hidden sequence The upsampling process is represented as:

[0086]

[0087] Where K is the length of the Mel spectrogram sequence, which is equal to N during the training phase, α is a hyperparameter controlling the extension time of each phoneme, and C... pho For phoneme hiding sequences, US is the upsampler. Let be the hidden sequence of the i-th Mel spectrum.

[0088] The specific process of phoneme duration conversion is as follows:

[0089] (1) The content information obtained after the content encoder, i.e., the Mel spectrum hidden sequence C melInput phoneme predictor.

[0090] (2) The phoneme predictor obtains the Mel spectrogram hidden sequence C. mel Corresponding phoneme tag sequence

[0091] (3) Hide the Mel spectrum sequence C mel Corresponding phoneme tag sequence Simultaneously, the input is downsampled in the downsampler to remove the speaker's phoneme duration information, resulting in the phoneme hidden sequence C. pho .

[0092] (4) Input the target speaker information obtained by the speaker encoder into the duration predictor to obtain the phoneme duration sequence of the target speaker.

[0093] (5) Hide the phoneme sequence C pho and the phoneme duration sequence of the target speaker Simultaneously, the input upsampler is used to upsample and add the phoneme duration information of the target speaker, resulting in the duration-converted Mel spectrogram hidden sequence. Speaker encoder:

[0094] To enable speaker embeddings to be applied to speech outside the training set speakers, a speaker encoder consisting of a 3-layer LSTM was pre-trained on a multi-speaker dataset using GE2E loss. Then, multiple speech samples from the same speaker were input into the speaker encoder. Finally, the average of the multiple embeddings generated by the speaker encoder was used as the speaker embedding. The overall architecture of the speaker encoder is as follows. Figure 5 As shown.

[0095] The specific process of the speaker encoder is as follows:

[0096] (1) Input the target speaker Mel spectrogram X into the speaker encoder. First, Mel spectrogram X is processed through three layers of LSTM to obtain X1.

[0097] (2) After passing through three layers of LSTM, extract the output of the last layer and denote it as X2.

[0098] (3) X2 passes through the Linear layer to obtain the speaker embedding representation S.

[0099] (4) In order to extract the same speaker embedding information from the same speaker, 10 speech samples of the same speaker are input into the speaker encoder to obtain 10 speaker embeddings, and the average value of the 10 speaker embeddings is taken as the speaker embedding representation of the current speaker.

[0100] Content encoder, decoder:

[0101] Models based on Transformer and Convolutional Neural Networks (CNNs) have achieved good results in speech recognition, outperforming recurrent neural networks. Transformer captures global interaction information within long sequences, while CNNs effectively utilize local features. Therefore, this paper proposes the Conformer model, a convolution-enhanced Transformer model, to model both local and global information in audio sequences for speech conversion.

[0102] The content encoder consists of a Conformer module, which converts the source speaker's Mel spectrogram into a sequence of hidden representations. The overall architecture of the content encoder is as follows: Figure 6 As shown, the model mainly consists of three parts: a feedforward module, a multi-head self-attention module, and a convolutional module, each using residuals. The feedforward module uses the Swish activation function and Dropout layers. The multi-head self-attention module uses relative position encoding, which makes the model more robust to various inputs of different lengths. The convolutional module consists of LayerNorm layers and one-dimensional convolutional layers with gated linear unit (GLU) activation. The decoder consists of a Conformer block and PostNet, which converts the hidden representation sequence and speaker embedding, after passing through a phoneme duration converter, into the Mel spectrogram of the target speaker. The PostNet then refines the Mel spectrogram. The PostNet structure is as follows... Figure 7 As shown.

[0103] The specific process of content encoder is as follows:

[0104] (1) Input the original speaker Mel spectrum X into the content encoder. First, it passes through the feedforward module, which includes LayerNorm, Linear, Swish activation and Dropout layers, to obtain X1=X+FFN(X), where FFN represents the feedforward module.

[0105] (2) After passing through the multi-head self-attention module, which includes the LayerNorm layer, the multi-head self-attention layer and the Dropout layer, we get X2 = X1 + MHA(X1), where MHA represents the multi-head self-attention module.

[0106] (3) X2 passes through a convolutional module to obtain X3 = X2 + Conv(X2), where Conv represents a convolutional module, including LayerNorm, 1D convolutional layer, gated linear unit layer, InstanceNorm layer and Dropout layer.

[0107] (4) After being connected to the speaker's embedded S, the data is simultaneously input into the feedforward module, which includes a LayerNorm layer, a Linear layer, a Swish activation layer, and a Dropout layer, to obtain... Where FFN represents the feedforward module. This indicates a Concat connection.

[0108] (5) Finally, the content embedding C is obtained through the LayerNorm layer.

[0109] The specific process of the decoder is as follows:

[0110] (1) Input the content information C after duration conversion and the target speaker information S into the content encoder. First, the content information passes through the feedforward module C, which includes LayerNorm, Linear, Swish activation and Dropout layers, to obtain C1 = C + FFN(C), where FFN represents the feedforward module.

[0111] (2) C1 passes through the multi-head self-attention module, which includes the LayerNorm layer, the multi-head self-attention layer and the Dropout layer, to obtain C2 = C1 + MHA(C1), where MHA represents the multi-head self-attention module.

[0112] (3) C2 is passed through a convolutional module to obtain C3 = C2 + Conv(C2), where Conv represents the convolutional module, including LayerNorm, 1D convolutional layer, gated linear unit layer, InstanceNorm layer and Dropout layer.

[0113] (4) After being connected to the speaker's embedded S, the data is simultaneously input into the feedforward module, which includes a LayerNorm layer, a Linear layer, a Swish activation layer, and a Dropout layer, to obtain... Where FFN represents the feedforward module. This indicates a Concat connection.

[0114] (5) C4 is transformed into the Mel spectrum after passing through LayerNorm and PostNet network. PN stands for PostNET network, which includes 1d convolutional layers and BatchNorm layers.

[0115] Self-reconfiguration process:

[0116] Suppose we have two Mel spectrogram sequences from different segments of the same speech, X and X'. u =[x u ,x u+1 ,…,x u+N ] and Xv =[x v ,x v+1 ,…,x v+N ], x u and x v This represents the start position of the same speech sound in different segments, and N represents the length of the Mel spectrogram. Traditional models mostly use X... u =X v The model is trained using X. Although X undergoes self-reconstruction based on its own features, this training mode differs from that of X for feature decomposition. u ≠X v The conversion settings at that time. Therefore, this invention performs self-reconstruction only on different segments within the same speech. The loss function used is defined as follows:

[0117] L recon =||X u -D(C u ,S v )||2+||X v -D(C v ,S u )||2 ⑸

[0118] Where X u and X v For the Mel spectra of the same speech in different segments, C u and C v For embedding representations of different speech segments, S v and S u Speaker representations are embedded for different segments of speech.

[0119] Since different loss functions can provide different underlying distributions, this invention uses a linear combination of different loss functions to better track the latent distribution of the input data, perform more reasonable self-reconstruction of the source speech, and better learn the distribution of the latent space. The self-reconstruction loss function for paired utterances is further modified by using a linear combination of L1 and L2 loss functions to calculate the difference between the output and the corresponding target. The modified loss function is shown below:

[0120]

[0121] α and β are hyperparameters, and α + β = 1 is always maintained.

[0122] The specific process of self-reconfiguration is as follows:

[0123] (1) C u and C v S v and S u Input to the decoder to obtain the Mel spectrogram sequence X after speaker embedding.u ′ and X v ′, where X u ′=D(C u ,S v ), X v ′=D(C v ,S u ), C u and C v For embedding representations of different speech segments, S v and S u Speaker embedding representations for different segments of speech, where D represents the decoder.

[0124] (2) The self-reconstruction loss L of a linear combination of L1 and L2 recon Mel spectrogram sequence X after speaker conversion u ′ and X v ′ and the corresponding original speaker Mel spectrum sequence X u and X v Perform content loss calculation.

[0125] Enhancing the consistency and separability of speaker style:

[0126] To ensure that speaker embeddings remain invariant for the same speaker, this invention adds a periodic consistency loss term. Assume x u ′ and x v ′ is the speech feature generated by the decoder, X u ′=D(C u ,S v ), X v ′=D(C v ,S u Then, the periodic consistency loss is defined as follows:

[0127] L cycle =||S u -S v ||2+||S u -E s (X u ′)||2+||X v -E s (X v ′)||2 ⑺

[0128] Where S u and S v E embeds speaker representations for different segments of speech. s For speaker encoders.

[0129] Periodic consistency loss makes speaker embeddings of the same speaker segment more compact. However, to maintain the discriminability of speaker embeddings in different speakers' speech, they must share mutual information with their corresponding utterances to achieve self-reconstruction. However, minimizing periodic consistency loss may also make speaker embeddings of different speakers less separable. Therefore, additional speaker identity information is used to address this issue. A projection vector is added on top of the speaker embedding for speaker classification. Let d represent the speaker's one-hot vector in the sample speech, and L... cross Representing cross-entropy loss, the speaker identity loss loss is expressed as:

[0130] L id =L cross (P r (S u ′),d)+L cross (P r (S v ′),d) ⑻

[0131] Where S u and S v P embeds speaker representations for different segments of speech. r To extract the speaker's embedded projection vector.

[0132] The specific process of style conversion is as follows:

[0133] (1) Before conversion, embed S in the speaker's name u and S v Add a projection vector at the very front to represent the speaker's identity information.

[0134] (2) Input the Mel spectrogram sequence after speaker conversion into the speaker encoder to obtain the converted speaker embedding representation, which is S u ′=E s (X u ′) and S v ′=E s (X v ′).

[0135] (3) Using periodic consistency loss L cycle Embedding S in the transformed speaker u ′ and S v Speaker embedding with the original speaker u and S v Perform loss calculation for the speaker.

[0136] (3) Extract the transformed speaker embedding S u ′ and S v The projection vector of ′, with identity loss L idEmbedding S in the transformed speaker u ′ and S v Speaker embedding with the original speaker u and S v Perform an identity loss calculation.

[0137] Conversion process:

[0138] The system proposed in this invention mainly consists of three parts. The first part separates speech content and speaker features using phoneme-level information bottlenecks and controls the duration of speaker phonemes through a phoneme duration converter. The second part trains a model using a modified self-reconstruction loss, while incorporating periodic consistency loss and identity loss to enhance speaker style consistency and separability, ensuring the separation of speech content and speaker features between different speakers. The trained model is used to synthesize the target's hidden sequence representation from the source speaker's hidden sequence representation. The third part uses a decoder composed of Conformers to quickly convert the transformed target hidden sequence into the target's Mel spectrogram. Finally, the transformed target Mel spectrogram is input into a vocoder to synthesize a speech waveform, obtaining the transformed speech, i.e., generating speech with the target speaker's identity information while retaining the source speaker's content.

[0139] The specific process of speech conversion is as follows:

[0140] (1) First, train the conversion model, obtain the speech database of the source speaker, and extract the Mel spectrogram sequences X of the source speaker from two different segments. u =[x u ,x u+1 ,…,x u+N ] and X v =[x v ,x v+1 ,…,x v+N [The following is a list of semantic features used for training, not a direct translation], where u and v are the starting indices of the Mel spectrogram sequence, and N is the length of the extracted Mel spectrogram sequence. The training process is as follows: Figure 8 As shown.

[0141] (2) The extracted Mel spectrum sequence is processed by the content encoder E c Encode the Mel spectrum into a Mel spectrum hidden sequence, represented as:

[0142] C u =E c (X u ) ⑼

[0143] C v =E c (X v ) 10

[0144] (3) Simultaneously input the source speaker's speech from these two different segments into the speaker encoder E. s Get speaker embedding S u and S v , is represented as:

[0145] S u =E s (X u ) ⑾

[0146] S v =E s (X v ) ⑿

[0147] (4) Embed the content of the corresponding segment and the speaker in C u S u With C v S v The input is fed into a duration converter, and the output is the duration-converted C. u ′ and C v ′.

[0148] (5) C u ′、S u C v ′、S v The input decoder utilizes the overall loss function L total Training is performed where, to refine the reconstructed Mel spectra after the initial estimation, PostNet is used at the end of the decoder to improve the quality of the generated Mel spectra. The overall loss function is shown below:

[0149] L total =L recon +λ p L pho +λ d L d +L cycle +L id 13

[0150] (6) After training, in the conversion phase, the original speaker’s speech Mel spectrogram X and the target speaker’s speech Mel spectrogram Y are obtained. The original speaker’s speech Mel spectrogram X is input into the content encoder to obtain the content embedding C. The target speaker’s speech Mel spectrogram Y is input into the speaker encoder to obtain the speaker embedding S.

[0151] (7) Input the content embedding C and the speaker embedding S into the phoneme duration converter at the same time to obtain the content embedding C′ after the phoneme duration conversion.

[0152] (8) Embed the content after phoneme duration conversion into C′ and speaker embedding into S simultaneously into the decoder to obtain the converted Mel spectrogram Y′.

[0153] (9) Input the converted Mel spectrogram Y′ obtained in (8) into the vocoder to synthesize the speech waveform and obtain high-quality speech similar to the target speaker.

Claims

1. A speech conversion method for personalized speech generation, characterized in that, include: Step 1: Train the conversion model, obtain the source speaker's speech database, and extract the source speaker's Mel spectrogram sequences X from two different segments. u =[x u ,x u+1 ,…,x u+N ] and X v =[x v ,x v+1 ,…,x v+N [, as speech features used for training;] Where u and v are the starting numbers of the Mel spectrum sequence, and N is the length of the extracted Mel spectrum sequence; Step 2: The extracted Mel spectrum sequence is processed by the content encoder E. c Encode the Mel spectrum into a Mel spectrum hidden sequence, represented as: C u =E c (X u ) ⑼ C v =E c (X v ) ⑽ Step 3: Simultaneously input the source speaker's speech from these two different segments into the speaker encoder E. s Get speaker embedding S u and S v , is represented as: S u =E s (X u ) ⑾ S v =E s (X v ) ⑿ Step 4: Embed the content of the corresponding segment and the speaker in C. u S u With C v S v Input into the phoneme duration converter, and convert C u The phoneme duration in the S v The duration of the phoneme, similarly for C v The phoneme duration in the S u The phoneme duration is calculated, and the output is the C after phoneme duration conversion. u ′ and C v ′; Step 5, set the corresponding C u ′、S u C v ′、S v The input decoder utilizes the overall loss function L total Training is performed where, to refine the reconstructed Mel spectra after the initial estimation, PostNet is used at the end of the decoder to improve the quality of the generated Mel spectra. The overall loss function is shown below: L total L recon +λ p L pho +λ d L d +L cycle +L id ⒀ Complete the training; Step 6, actual conversion: obtain the original speaker's speech Mel spectrogram X and the target speaker's speech Mel spectrogram Y. Input the original speaker's speech Mel spectrogram X into the content encoder to obtain the content embedding C. Input the target speaker's speech Mel spectrogram Y into the speaker encoder to obtain the speaker embedding S. Step 7: Simultaneously input the content embedding C and the speaker embedding S into the phoneme duration converter to obtain the content embedding C′ after phoneme duration conversion; Step 8: Simultaneously input the converted phoneme duration content into C′ and the speaker embedding S into the decoder to obtain the converted Mel spectrogram Y′; Step 9: Input the converted Mel spectrogram Y′ obtained in step 8 into the vocoder to synthesize the speech waveform and obtain high-quality speech similar to the target speaker.

2. The speech conversion method for personalized speech generation according to claim 1, characterized in that, Step 2 includes the following sub-steps: Sub-step A1: Hide the Mel spectrum sequence C mel Input phoneme predictor; Sub-step A2: Use the phoneme predictor to obtain the Mel spectrogram hidden sequence C. mel Corresponding phoneme tag sequence Sub-step A3, hide the Mel spectrum sequence C mel Corresponding phoneme tag sequence Simultaneously, the input is downsampled in the downsampler to remove the speaker's phoneme duration information, resulting in the phoneme hidden sequence C. pho ; Sub-step A4: Input the target speaker information obtained by the speaker encoder into the duration predictor to obtain the phoneme duration sequence of the target speaker. Sub-step A5, hide the phoneme sequence C pho and the phoneme duration sequence of the target speaker Simultaneously, the input upsampler is used to upsample and add the phoneme duration information of the target speaker, resulting in the duration-converted Mel spectrogram hidden sequence.

3. The speech conversion method for personalized speech generation according to claim 2, characterized in that, The phoneme predictor P pho Used to predict C mel phoneme tag sequence The phoneme tag sequence obtained by optimization prediction With the real phoneme label sequence P = [p1, p2, ..., p N The phoneme predictor is trained using the phoneme classification loss between [ ], which is expressed as: Where N is the length of the Mel spectrum of the hidden sequence, P pho This represents a phoneme predictor; Given a predicted phoneme hiding sequence Then, the downsampler DS is used to sample C. mel Perform downsampling; the downsampling process is represented as follows: Where C mel Hidden sequences for Mel spectra, Given a sequence of predicted phoneme labels, DS is a downsampler. Let M be the hidden sequence of the i-th phoneme, and M be the length of the hidden sequence of the phoneme. In order to hide the phoneme sequence C pho Expand back to hidden sequence C mel The duration predictor P was used. d and a phoneme-level duration-based upsampler US; In order to predict C using information from the target speaker pho The corresponding phoneme duration sequence, C after downsampling pho Both the speaker embedding S and the duration predictor P are input into the duration predictor. d middle; P d The prediction corresponds to C pho Phoneme duration sequence By embedding the source speaker into P d To predict source speaker information The duration sequence of the actual phonemes is D = [d1, d2, ..., d]. M The duration predictor is trained using the L1 loss between [ ], and the loss function is expressed as: Where M is the length of the phoneme hidden sequence, P d For duration predictor, Let d be the hidden sequence of the i-th phoneme. i The duration of the i-th phoneme; Upsampler US will C pho Expanded to be based on Extended Mel spectrum hidden sequence The upsampling process is represented as: Where K is the length of the Mel spectrogram sequence, which is equal to N during the training phase, α is a hyperparameter controlling the extension time of each phoneme, and C... pho For phoneme hiding sequences, US is the upsampler. For the hidden sequence of the i-th Mel spectrum, It is a sequence of phoneme durations.

4. The speech conversion method for personalized speech generation according to claim 1, characterized in that, The human encoder E described in step 3 s The encoding process includes the following sub-steps: Sub-step B1: Input the target speaker Mel spectrogram X into the speaker encoder. The Mel spectrogram X is processed through three layers of LSTM to obtain X1. Sub-step B2 extracts the last layer output of sub-step B1, denoted as X2; Sub-step B3, X2 passes through the Linear layer to obtain the speaker embedding representation S. In sub-step B4, in order to extract the same speaker embedding information from the same speaker, 10 speech samples from the same speaker are input into the speaker encoder to obtain 10 speaker embeddings, and the average value of the 10 speaker embeddings is taken as the speaker embedding representation of the current speaker.

5. The speech conversion method for personalized speech generation according to claim 1, characterized in that, The specific processing of the content encoder includes the following sub-steps: Sub-step C1: Input the original speaker Mel spectrogram X into the content encoder, and pass it through the feedforward module, which includes the LayerNorm layer, the Linear layer, the Swish activation layer and the Dropout layer, to obtain X1 = X + FFN(X), where FFN represents the feedforward module; Sub-step C2, after passing through the multi-head self-attention module, including the LayerNorm layer, the multi-head self-attention layer and the Dropout layer, yields X2 = X1 + MHA(X1), where MHA represents the multi-head self-attention module; In sub-step C3, X2 passes through a convolutional module to obtain X3 = X2 + Conv(X2), where Conv represents the convolutional module, which includes LayerNorm, 1D convolutional layer, gated linear unit layer, InstanceNorm layer and Dropout layer; Sub-step C4 involves connecting the speaker's embedded S and simultaneously inputting it into the feedforward module to obtain... Indicates a Concat connection; Sub-step C5, after passing through the LayerNorm layer, yields the content embedding C.

6. A speech conversion method for personalized speech generation according to claim 1 or 5, characterized in that, The specific processing of the decoder includes the following sub-steps: Sub-step D1: Input the content information C after duration conversion and the target speaker information S into the content encoder. The content information passes through the C feedforward module, which includes the LayerNorm layer, Linear layer, Swish activation layer and Dropout layer, to obtain C1 = C + FFN(C), where FFN represents the feedforward module. In sub-step D2, C1 passes through the multi-head self-attention module, which includes the LayerNorm layer, the multi-head self-attention layer, and the Dropout layer, to obtain C2 = C1 + MHA(C1), where MHA represents the multi-head self-attention module; In sub-step D3, C2 passes through a convolutional module to obtain C3 = C2 + Conv(C2), where Conv represents the convolutional module, which includes LayerNorm, 1D convolutional layer, gated linear unit layer, InstanceNorm layer, and Dropout layer. Sub-step D4 involves connecting the speaker's embedded S and simultaneously inputting it into the feedforward module to obtain... in Indicates a Concat connection; Sub-steps D5 and C4 pass through LayerNorm and PostNet to obtain the transformed Mel spectrum. PN stands for PostNET network, which includes 1d convolutional layers and BatchNorm layers.

7. The speech conversion method for personalized speech generation according to claim 1, characterized in that, Step 6 also includes style transfer, which comprises the following sub-steps: Sub-step E1, embedding S in the speaker before conversion u and S v Add a projection vector at the very beginning to represent the speaker's identity information; Sub-step E2 involves inputting the converted speaker Mel spectrogram sequence into the speaker encoder to obtain the converted speaker embedding representation, which is S. u ′=E s (X u ′) and S v ′=E s (X v ′); Sub-step E3, with periodic consistency loss L cycle Embedding S in the transformed speaker u ′ and S v Speaker embedding with the original speaker u and S v Calculate the speaker's losses; Sub-step E4: Extract the transformed speaker embedding S u ′ and S v The projection vector of ′, with identity loss L id Embedding S in the transformed speaker u ′ and S v Speaker embedding with the original speaker u and S v Perform an identity loss calculation.

Images