Cross-lingual speech conversion method based on mutual information and SE attention mechanism

Abstract

The present application relates to the technical field of speech conversion, in particular to a cross-lingual speech conversion method based on mutual information and SE attention mechanism, comprising a training stage and a conversion stage, the training stage comprising the following steps: first, introducing an SE module in the content encoder, through which the network can learn to obtain global information and improve the quality of the content representation generated by the content encoder; further introducing mutual information as a correlation measure to reduce the dependency between the content representation, speaker representation and fundamental frequency feature in an unsupervised manner, thereby achieving efficient disentanglement; finally, reconstructing the individual information in the target speaker sentence and the content information in the source speaker sentence in the decoder, thereby achieving high-quality cross-lingual speech conversion. At the same time, the method can also convert speakers not in the training set, i.e. complete the cross-lingual speech conversion of any speaker in an open set.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of speech conversion, in particular to a cross-lingual speech conversion method based on mutual information and SE attention mechanism. BACKGROUND

[0002] Speech conversion is an important branch of research in the field of speech signal processing, and the research of this technology has important theoretical value and application prospect. Speech conversion is a technology that converts the identity features of the source speaker in the speech to the identity features of the target speaker while ensuring that the semantic features in the speech remain unchanged. In short, it is to convert a person's speech to sound like it is emitted by another specified speaker.

[0003] With the booming development of voice conversion technology, Abe et al. proposed the cross-lingual voice conversion problem in 1990. Traditional voice conversion only focuses on solving the same language conversion problem, that is, the source and target speaker voices are required to be of the same language. However, in the research of cross-lingual voice conversion, the source and target speakers speak different languages and texts, and there are significant differences in phonemes, tones and stress in different languages. Therefore, the research of cross-lingual voice conversion is more difficult. The initial method of realizing cross-lingual voice conversion has a high requirement for training corpus, which requires the same speaker to provide two sets of corpus of different languages, that is, it depends on the parallel data set of bilingual speakers. However, it is difficult to obtain voice data, and it is more difficult and unrealistic to obtain a parallel text data set of bilingual speakers. In 2003, Kumar et al. proposed a cross-lingual voice conversion method based on vocal tract length normalization (VTLN). This method constructs a regularization function between the phonemes or acoustic categories of the two language systems, but the size of the conversion model is limited, and only the conversion of gender features can be realized. In a long period of time before and after 2010, many alignment methods for finding source-target frame pairs from non-parallel text data sets emerged, such as unit selection and iterative frame alignment methods. However, due to the large differences between phonemes in different languages, the alignment of cross-lingual voice conversion inevitably introduces errors, thereby affecting the performance of the system. In 2018, representation learning was used to solve the cross-lingual voice conversion problem. Mohammadi et al. proposed a factorized hierarchical variational autoencoder (FHVAE). Although this method is no longer limited to parallel data sets of bilingual speakers, the conversion of the voice after the smoothing effect of the variational autoencoder (VAE) causes obvious noise in the converted voice. In 2019, the phonetic posterior gram (PPG) was applied to cross-lingual voice conversion, but this method directly depends on the performance of the speech recognition system involved.

[0004] Most of the above-mentioned voice conversion methods can only realize voice conversion in a closed set, that is, the source and target speakers need to participate in the training during the model training process. However, in actual application scenarios, only a small number or a sentence of corpus may participate in the training for the source or target speaker, or even no training is involved. Such problems belong to the category of voice conversion in an open set, which is more challenging. SUMMARY

[0005] To solve the above problems, the application provides a cross-lingual speech conversion method based on mutual information and SE attention mechanism, which uses mutual information as a correlation measure, reduces the dependency between content representation, speaker representation and fundamental frequency feature in an unsupervised manner, and realizes efficient disentanglement; further, an SE module is introduced in the content encoder, the SE module compresses local features into a single global context vector weight, and the output of the SE module is generated by multiplying the global context vector weight and the feature element, so that better content representation is obtained, the quality of the converted speech is improved, and finally high-quality cross-lingual speech conversion of any speaker in an open set is realized.

[0006] The specific technical solutions adopted by the application are as follows:

[0007] A cross-lingual speech conversion method based on mutual information and SE attention mechanism, comprising a training phase and a conversion phase:

[0008] The training phase comprises the following steps:

[0009] Step 1, a large amount of training corpus is obtained, the training corpus is composed of sentences of multiple speakers of different languages;

[0010] Step 2, the obtained training corpus is preprocessed and feature extracted, 128 frames are randomly selected from the mel spectrum feature of each training corpus, and a mel spectrum feature segment is obtained through cascading operation, which is represented as acoustic feature X;

[0011] Step 3, input the acoustic feature into the conversion network for training, the conversion network is composed of a content encoder, a speaker encoder, a pitch extractor and a decoder; wherein the content encoder and the speaker encoder are used to disentangle the speaker's speech and obtain the content information and speaker information of the sentence, the pitch extractor is used to extract the fundamental frequency F0 from the waveform of the speaker's speech, and the decoder is used to reconstruct the obtained content information, speaker information and fundamental frequency F0 to obtain the reconstructed mel spectrum feature, which is represented as acoustic feature X';

[0012] Step 4, during the training process of the conversion network, the reconstruction loss function of the conversion network is as small as possible, the hyperparameters of the conversion network are set to minimize the objective function, until the set number of iterations, so as to obtain the trained conversion network;

[0013] The conversion phase comprises the following steps:

[0014] Step 5, select speech of different languages as test corpus, and extract source speaker mel spectrum feature X1 and target speaker mel spectrum feature X2 in the test corpus as acoustic features;

[0015] Step 6: Input the acoustic features X1 and X2 into the trained conversion network to reconstruct the Mel-spectral features X of the converted statement. 1→2 ;

[0016] Step 7: Using a Parallel WaveGAN vocoder, the Mel spectrum features of the obtained converted sentences are converted into speech for output, thereby obtaining synthesized speech for cross-language speech conversion;

[0017] In the above technical solution, the conversion network consists of a content encoder, a speaker encoder, a pitch extractor, and a decoder. The content encoder consists of one h-net network, one quantizer q, and one g-net network. The h-net network consists of one one-dimensional standard convolutional layer with a stride of 2, four convolutional modules, and one 512-dimensional linear layer. There are four convolutional modules in total. The first three convolutional modules consist of layer normalization, a 512-dimensional linear layer, and a ReLU activation function. The last convolutional module consists of layer normalization, a 512-dimensional linear layer, an SE module, and a ReLU activation function. The SE module consists of one one-dimensional global average pooling layer, two fully connected layers, one ReLU activation function, one Sigmoid activation function, and dense residual connections.

[0018] A further improvement to the present invention is that the training process in step 4 is as follows:

[0019] 4.1. The speaker's Mel spectral feature sequence X k ={x k,1 ,x k,2 ,…,x k,T The input to the h-net network of the content encoder yields the dense feature sequence Z. k ={z k,1 ,z k,2 ,…,z k,T / 2 The sequence length changes from T to T / 2;

[0020] 4.2. The dense feature sequence Z k ={z k,1 ,z k,2 ,…,z k,T / 2 The input is fed into the SE module, which consists of one one-dimensional standard average pooling layer, two fully connected layers, one ReLU activation function, one Sigmoid activation function, and dense residual connections. The two fully connected layers are alternated with the ReLU and Sigmoid activation functions. The SE module compresses local features into a single global context vector weight, and multiplies the global context vector weight with the feature elements to obtain the dense feature sequence Z. k Reweight and generate the output Z′ of the SE module k ={z′ k,1 ,z′k,2 ,…,z′ k,T / 2};

[0021] 4.3 The dense feature sequence Z' after reweighting the SE module k ={z' k,1 ,z' k,2 ,…,z' k,T / 2 The input is fed into quantizer q, which performs quantization based on the trainable codebook B, converting Z' into its input. k ={z' k,1 ,z' k,2 ,…,z' k,T / 2 Discretization yields dense feature sequences. It is the closest to Z' in the trainable codebook B. k The vector, vector quantization removes Z' by setting an information bottleneck. k Non-essential information in the text makes It is associated with the underlying speech information, and then the content encoder is trained by minimizing the loss function of the quantizer q;

[0022] 4.4. Dense feature sequences The input is fed into the g-net network to generate the aggregate sequence R. k ={r k,1 ,r k,2 ,…,r k,T / 2}, given r k,t Contrastive predictive coding is employed, which distinguishes positive samples after m steps by minimizing the loss function of the g-net network. And negative samples extracted from the negative sample set Ω, by predicting future samples with probabilistic contrastive loss, encode local features spanning multiple time steps into In the middle, As a representation of content;

[0023] 4.5. The speaker's Mel spectral feature sequence X k ={x k,1 ,x k,2 ,…,x k,T The input is fed into the speaker encoder to generate the speaker feature vector S. k , will S k As a representation of the speaker;

[0024] 4.6. The speaker's Mel spectral feature sequence X k ={x k,1 ,x k,2 ,…,x k,T The input is fed into a pitch extractor to extract the fundamental frequency F0, and F0 is logarithmically normalized to generate a speaker-independent fundamental frequency feature sequence P. k ={pk,1 ,p k,2 ,…,p k,T};

[0025] 4.7 Representing Content Speaker representation S k and fundamental frequency characteristic sequence P k ={p k,1 ,p k,2 ,…,p k,T The input is fed into the decoder for training, minimizing the decoder's loss function;

[0026] 4.8 Representation of Calculation Content Speaker representation S k and fundamental frequency characteristic sequence P k The mutual information between each of the three factors is multiplied by a constant weight and added to the total loss of speech conversion, and then the total loss of speech conversion is minimized.

[0027] 4.9 Repeat steps 4.1 to 4.8 until the set number of iterations is reached, thus obtaining the trained network.

[0028] A further improvement to the present invention is that, in the above technical solution, the input process in step 6 is as follows:

[0029] 6.1 Input the Mel-spectral feature X1 of the source speaker into the content encoder and pitch extractor to generate content representations respectively. Fundamental frequency eigenvector P' k The Mel-spectral features X2 of the target speaker are input into the speaker encoder to generate the speaker representation S'. k ;

[0030] 6.2 Representing Content Speaker representation S' k and fundamental frequency eigenvector P' k The input is fed into the decoder to generate the converted Mel-spectral features X. 1→2 .

[0031] In this invention, the total loss of the speech conversion model is:

[0032] L VC =L VQ +L CPC +L REC +λ MI L MI

[0033] Where, λ MI ≥0 represents the regularization parameter for mutual information loss, L VQ L CPC L REC, L MI respectively represent the vector quantization loss of quantizer q, the information noise contrast estimation loss of contrastive predictive coding, the reconstruction loss of decoder and the mutual information quantity loss.

[0034] The loss function of quantizer q is:

[0035]

[0036] where K represents the number of training sentences, T represents the fixed frame number of mel-spectral features of each training sentence, z' k,t represents the dense feature sequence obtained after the SE module is reweighted, represents the dense feature sequence after trainable codebook quantization, sg(·) represents the stop gradient operator, represents the square of 2-norm.

[0037] The loss function of contrastive predictive coding is:

[0038]

[0039] where K represents the number of training sentences, T represents the fixed frame number of mel-spectral features of each training sentence, M represents the step of contrastive prediction, T' = T / 2-M, exp(·) represents the (·) power of e, W m represents a trainable projection matrix, Ω k,t,m is a negative sample set composed of randomly selected samples from the current training sentence, represents a random sample in the negative sample set Ω k,t,m , R k ={r k,1, r k,2 ,…,r k,T / 2} represents a dense feature sequence The aggregated sequence output after a 256-dimensional one-way recurrent neural network, r k,t represents a sample of the aggregated sequence R k .

[0040] The reconstruction loss of the decoder is:

[0041]

[0042] where K represents the number of training sentences, T represents the fixed frame number of mel-spectral features of each training sentence, x t represents the input mel-spectral feature segment, represents the reconstructed spectral feature of the decoder, ||·||1 represents the 1-norm, and ||·||2 represents the 2-norm.

[0043] The mutual information quantity loss is:

[0044]

[0045] in, These represent unbiased estimates of the upper bounds of the variational contrast logarithms between content representation and speaker representation, content representation and fundamental frequency feature, and fundamental frequency feature and speaker representation, respectively.

[0046] In this invention, the unbiased estimate of the upper bound of the variational contrast logarithm between content representation and speaker representation is:

[0047]

[0048] Where K represents the number of training statements, and T represents the number of fixed frames for the Mel-spectral features of each training statement. The conditional log-likelihood of a positive sample pair (speaker representation, content representation) is represented by... The conditional log-likelihood of negative sample pairs (speaker representation, content representation) is expressed as... When representing a speaker's representation S, after... Content representation after network parameterization The corresponding variational approximation of the true posterior. This represents the content representation of the t-th frame corresponding to the k-th sentence in a positive sample pair. s represents the content representation of the t-th frame of the l-th sentence in the negative sample pair. k This represents the speaker representation of the k-th sentence in a positive sample pair.

[0049] The unbiased estimate of the upper bound of the variational contrast logarithm between content representation and fundamental frequency features is:

[0050]

[0051] Where K represents the number of training statements, and T represents the number of fixed frames for the Mel-spectral features of each training statement. The conditional log-likelihood of a positive sample pair (fundamental frequency feature, content representation) is represented by... The conditional log-likelihood of negative sample pairs (fundamental frequency features, content representations) is represented by... This indicates that, given a fundamental frequency characteristic P, after... Content representation after network parameterization The corresponding variational approximation of the true posterior. This represents the content representation of the t-th frame corresponding to the k-th sentence in a positive sample pair. This represents the content representation of the t-th frame corresponding to the l-th sentence in the negative sample pair.

[0052] The unbiased estimate of the upper bound of the variational contrast logarithm between fundamental frequency features and speaker representations is:

[0053]

[0054] where K denotes the number of training sentences, T denotes the fixed frame number of mel-spectrogram feature of each training sentence, denotes the conditional log-likelihood of the positive sample pair (speaker representation, pitch feature), denotes the conditional log-likelihood of the negative sample pair (speaker representation, pitch feature), denotes the true posterior of the pitch feature P given the speaker representation S, p P,S denotes the variational approximation of the true posterior of the pitch feature P given the speaker representation S, p k,t denotes the pitch feature of the t-th frame of the k-th sentence in the positive sample pair, s k denotes the speaker representation of the k-th sentence in the positive sample pair, p l,t denotes the pitch feature of the t-th frame of the l-th sentence in the negative sample pair.

[0055] The beneficial effects of the present application: compared with the prior art, the model in the present application realizes cross-lingual speech conversion by learning the disentanglement ability of the speech, integrates the mutual information loss into the model training process, the mutual information is the Kullback-Leibler divergence (Kullback-Leibler divergence) of the joint distribution and the edge distribution between the content representation, the speaker representation and the fundamental frequency feature, the upper bound of the mutual information is calculated by using the variational contrast logarithmic upper bound, the mutual dependence is reduced by minimizing the unbiased estimation of the variational contrast logarithmic upper bound between the content representation, the speaker representation and the fundamental frequency feature, and the efficient disentanglement is realized, so as to improve the quality of the converted speech; the SE module is introduced into the last convolution block of the encoder in the present application, the dense feature sequence input into the SE module is subjected to a global average pooling operation, so that the dense feature sequence is aggregated in the spatial dimension, a channel descriptor and a global distribution embedding corresponding to the channel feature are generated, all layers are allowed to use, after aggregation, a simple gating mechanism and a Sigmoid function are used, the generated global distribution embedding corresponding to the channel feature is used as input, and a set of channel modulation weights is generated, the weights are multiplied with the feature elements to form a new feature sequence as the output of the SE module, at this time, the global context information is contained in the feature sequence, and better content representation can be generated, and the quality of the converted speech is improved; the SE module can enable the convolutional neural network (CNN) to obtain global information, so that the content encoder can generate high-quality content representation, by introducing the mutual information as a correlation measure and then minimizing the mutual information loss, the dependence between the content representation, the speaker representation and the fundamental frequency feature can be reduced, and efficient disentanglement is realized. Through the above two innovations, the method can realize high-quality cross-lingual speech conversion. Because the introduction of the mutual information enables the model to realize efficient disentanglement, in the conversion stage, for any new source or target speaker, the model can obtain the content representation, the speaker representation and the fundamental frequency feature through the above steps, and the speech conversion can be realized only by replacing the speaker representation of the source speaker with the speaker representation of the target speaker, that is, the model in the present application is also applicable to the speech conversion task in the open set situation. Therefore, the present application is a cross-lingual arbitrary speaker speech conversion method with high synthesized sound quality and individual similarity. BRIEF DESCRIPTION OF DRAWINGS

[0056] Figure 1 is a principle schematic diagram of the model described in the embodiments of the present application.

[0057] Figure 2 is a network structure diagram of the content encoder in the model described in the embodiments of the present application.

[0058] Figure 3is a network structure diagram of a speaker encoder in the model described in the embodiment of the application.

[0059] Figure 4 is a network structure diagram of a decoder in the model described in the embodiment of the application. DETAILED DESCRIPTION

[0060] The technical solutions in the embodiments of the application will be clearly and completely described below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, not all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor fall within the protection scope of the application.

[0061] Embodiment: A cross-lingual speech conversion method based on mutual information and SE attention mechanism, including a training stage and a conversion stage, the training stage is used to obtain the conversion network and its parameters required for speech conversion, and the conversion stage is used to realize the conversion of the individual information of the source speaker voice into the individual information of the target speaker voice while keeping the semantic content of the source speaker voice unchanged.

[0062] As shown in Figure 1 , the training stage includes the following steps:

[0063] Step 1, obtain training corpus, the training corpus comes from AISHELL-3 Chinese corpus library and CSTR VCTK English corpus library; from the obtained Chinese and English corpus library, 40 speakers in the AISHELL-3 Chinese corpus library are selected, each speaker has about 200 sentences, and each sentence has a duration of 2-10s; 40 speakers in the CSTR VCTK English corpus library are selected, each speaker has about 200 sentences, and each sentence has a duration of 2-8s; the selected sentences of Chinese and English are jointly used to form training corpus for model training, so that the conversion network can fully learn the pronunciation characteristics and semantic information of different languages, thereby ensuring the performance of the conversion model;

[0064] Step 2, obtain acoustic features, pre-process and feature extraction are performed on the obtained training corpus, 128 frames are randomly selected from the mel spectrum features of each training corpus, and a mel spectrum feature segment is obtained through cascading operation, which is represented as acoustic feature X;

[0065] Step 3, input the acoustic feature X obtained in step 2 into the conversion network for training, the network in the present example adopts a double-encoder structure, the dependency between the content representation, the speaker representation and the fundamental frequency feature of the input sentence is reduced by introducing and minimizing the mutual information loss in the training process, efficient disentanglement is realized, the content representation, the speaker representation and the fundamental frequency feature are obtained, the content representation, the speaker representation and the fundamental frequency feature are input into the decoder for decoding, and cross-lingual speech conversion is realized;

[0066] The network mainly includes four parts of a content encoder, a speaker encoder, a pitch extractor and a decoder:

[0067] (1) In the content encoder, the h-net network encodes the input mel-frequency spectrum feature X k into a dense feature sequence Z k , which is input into an SE module, the dense feature sequence Z k is aggregated in the spatial dimension after a global average pooling operation to generate a channel descriptor and an embedding of a global distribution corresponding to the channel feature, which is allowed to be used by all layers, after aggregation, a simple gating mechanism and a Sigmoid function are used to generate the embedding of the global distribution corresponding to the channel feature as input and produce a set of channel modulation weights, multiply these weights with the feature elements to generate a new feature sequence Z' k As the output of the SE module, then the quantizer q sets an information bottleneck through vector quantization to remove unnecessary information in the feature sequence Z' k , so as to associate it with the underlying speech information, realize disentanglement, and generate a feature sequence The g-net network takes the feature sequence as input, encodes the local features into by predicting future samples with a probability contrast loss, and takes as the content representation. The content encoder structure is shown in Figure 2 . The content encoder is composed of 1 h-net network, 1 quantizer q and 1 g-net network, wherein the h-net network is composed of 1 one-dimensional standard convolution layer with a step of 2, 4 convolution modules and 1 512-dimensional linear layer, the convolution modules are 4 in total, the first 3 convolution modules are composed of layer normalization, a 512-dimensional linear layer and a ReLU activation function, the last convolution module is composed of layer normalization, a 512-dimensional linear layer, an SE module and a ReLU activation function, the SE module is composed of 1 one-dimensional global average pooling layer, 2 fully connected layers, 1 ReLU activation function, 1 Sigmoid activation function and a dense residual connection, the quantizer q includes 1 codebook, the codebook has 512 64-dimensional learnable vectors, and the g-net is a 256-dimensional one-way recurrent neural network layer.

[0068] The loss function of the quantizer q is:

[0069]

[0070] Where K represents the number of training statements, T represents the number of fixed frames for the Mel-spectral features of each training statement, and z' k,t This represents the dense feature sequence obtained after reweighting by the SE module. This represents the dense feature sequence after quantization of the trainable codebook, and sg(·) represents the stopping gradient operator. This represents the square of the 2-norm. In this example, K is 16000 and T is 128.

[0071] The loss function for contrastive predictive coding is:

[0072]

[0073] Where K represents the number of training statements, T represents the fixed number of frames for the Mel-spectral features of each training statement, M represents the step size for contrastive prediction, T' = T / 2 - M, exp(·) represents e raised to the power of (·), and W m Ω represents the trainable projection matrix. k,t,m It is a set of negative samples randomly selected from the current training statement. Ω represents the set of negative samples k,t,m Random samples in R k ={r k,1 ,r k,2 ,…,r k,T / 2} represents a dense feature sequence The aggregated sequence output after passing through a 256-dimensional unidirectional recurrent neural network, r k,t Represents the aggregate sequence R k One sample. In this example, K is 16000, T is 128, M is 6, and the negative sample set Ω k,t,m The sample size is 10.

[0074] (2) In the speaker encoder, the Mel spectral feature sequence X k As input, a feature vector S is generated. k As a speaker representation. The speaker encoder, such as... Figure 3 As shown, the speaker encoder consists of 8 ConvBank layers and 12 convolutional layers. Each convolutional layer consists of 1 average pooling layer and 4 linear layers. The ConvBank layers are used to encode long-term information, and the convolutional layers are used to derive a 256-dimensional speaker representation.

[0075] (3) In the pitch extractor, since the fundamental frequency feature is expected to contain the prosody variation but not the content information and speaker information, the F0 is directly extracted from the waveform and z-normalized independently for each speech. In the present example, the logF0 is adopted as the fundamental frequency feature P k Since P k is speaker-independent, the speaker information will rely on the speaker encoder to provide.

[0076] (4) In the decoder, the input content representation, speaker representation and fundamental frequency feature are fused and decoded to generate the reconstructed spectral feature highly similar to the input spectral feature. The decoder is shown in Fig. 2. Figure 4 The decoder consists of one 1024-dimensional bidirectional long short-term memory network layer, three convolutional layers, two 1024-dimensional bidirectional long short-term memory network layers and one 80-dimensional linear layer.

[0077] The reconstruction loss of the decoder is:

[0078]

[0079] where K represents the number of training sentences, T represents the fixed frame number of the mel-spectral feature of each training sentence, x t represents the input mel-spectral feature segment, represents the spectral feature reconstructed by the decoder, ||·||1 represents the 1-norm, and ||·||2 represents the 2-norm. In the present example, K is 16000 and T is 128.

[0080] (5) In the model training process, the mutual information is introduced as a correlation measure, and the mutual information loss is added to the total loss of the voice conversion to reduce the dependency between the content representation, speaker representation and fundamental frequency feature in an unsupervised manner, and to realize efficient disentanglement.

[0081] The total loss of the voice conversion model is:

[0082] L VC = L VQ + L CPC + L REC + λ MI L MI

[0083] where λ MI ≥ 0 represents the regularization parameter of the mutual information loss, L VQ , L CPC , L REC , L MIrespectively represent the vector quantization loss of quantizer q, the information noise contrast estimation loss of contrastive predictive coding, the reconstruction loss of the decoder, and the mutual information loss. In the present example, λ MI is 0.1.

[0084] The mutual information loss is:

[0085]

[0086] wherein, respectively represent the unbiased estimates of the variational contrastive log upper bound between the content representation and the speaker representation, the content representation and the fundamental frequency feature, and the fundamental frequency feature and the speaker representation.

[0087] The unbiased estimate of the variational contrastive log upper bound between the content representation and the speaker representation is:

[0088]

[0089] wherein, K represents the number of training sentences, T represents the fixed frame number of the mel-spectrum feature of each training sentence, represents the conditional log-likelihood of the positive sample pair (speaker representation, content representation), represents the conditional log-likelihood of the negative sample pair (speaker representation, content representation), represents the content representation after the network is parameterized, given the speaker representation S, represents the variational approximation of the corresponding true posterior, represents the content representation corresponding to the t-th frame of the k-th sentence in the positive sample pair, represents the content representation corresponding to the t-th frame of the l-th sentence in the negative sample pair, s k represents the speaker representation of the k-th sentence in the positive sample pair. In the present example, K is 16000, and T is 128.

[0090] The unbiased estimate of the variational contrastive log upper bound between the content representation and the fundamental frequency feature is:

[0091]

[0092] wherein, K represents the number of training sentences, T represents the fixed frame number of the mel-spectrum feature of each training sentence, represents the conditional log-likelihood of the positive sample pair (fundamental frequency feature, content representation), represents the conditional log-likelihood of the negative sample pair (fundamental frequency feature, content representation), represents the content representation after the network is parameterized, given the fundamental frequency feature P, ​​a variational approximation of the true posterior, represents the content representation corresponding to the t-th frame of the k-th sentence in the positive sample pair, represents the content representation corresponding to the t-th frame of the l-th sentence in the negative sample pair. In this example, K is 16000, and T is 128.

[0093] The unbiased estimation of the variational contrastive log upper bound between the fundamental frequency feature and the speaker representation is:

[0094]

[0095] where K represents the number of training sentences, and T represents the fixed frame number of the mel spectrum feature of each training sentence, represents the conditional log-likelihood of the positive sample pair (speaker representation, fundamental frequency feature), represents the conditional log-likelihood of the negative sample pair (speaker representation, fundamental frequency feature), represents the fundamental frequency feature P corresponding to the given speaker representation S when passing through the θ P,S The variational approximation of the true posterior corresponding to the network parameterized fundamental frequency feature P is p k,t represents the fundamental frequency feature corresponding to the t-th frame of the k-th sentence in the positive sample pair, k represents the speaker representation corresponding to the k-th sentence in the positive sample pair, l,t represents the fundamental frequency feature corresponding to the t-th frame of the l-th sentence in the negative sample pair. In this example, K is 16000, and T is 128.

[0096] Step 3-1, input the acoustic feature X extracted in step 2 into the content encoder, the speaker encoder and the pitch extractor to generate the content representation the speaker representation S and the fundamental frequency feature P;

[0097] Step 3-2, input the content representation the speaker representation S and the fundamental frequency feature P into the decoder, pass through the decoding network, and reconstruct the spectrum feature X', in the training process, the smaller the total loss of the above voice conversion, that is, the more similar the reconstructed spectrum is to the real spectrum.

[0098] Step 4, repeat steps 3-1 to 3-2 until the set number of iterations is reached, thereby obtaining the trained network. Due to the difference in specific settings of the neural network and the difference in performance of the experimental equipment, the set number of iterations is also different. In this experiment, the number of iterations is set to 100000 times;

[0099] The conversion stage includes the following steps:

[0100] Step 5, obtain a test corpus, the test corpus is from the AISHELL-3 Chinese corpus and the CSTR VCTK English corpus; from the obtained Chinese and English corpus, 8 speakers in the AISHELL-3 Chinese corpus are randomly selected, each speaker has about 200 sentences, 10 sentences are randomly selected from the 200 sentences, and the time length of each sentence is 2-10s; 8 speakers in the CSTR VCTK English corpus are selected, each speaker has about 200 sentences, 10 sentences are randomly selected from the 200 sentences, and the time length of each sentence is 2-8s. The selected sentences of the two languages of Chinese and English are jointly composed into a test corpus, and the source speaker mel spectrum feature X1 and the target speaker mel spectrum feature X2 of the test corpus are extracted as acoustic features;

[0101] Step 6, input the acoustic features into the trained conversion network to reconstruct the mel spectrum features of the converted sentence;

[0102] Step 6-1, input the acoustic features X1 extracted in step 5 into the trained content encoder and pitch extractor to generate the content representation of the source speaker and the fundamental frequency feature P', and input the acoustic features X2 extracted in step 5 into the trained speaker encoder to generate the speaker representation S' of the target speaker;

[0103] Step 6-2, input the content representation of the source speaker the fundamental frequency feature P' and the speaker representation S' of the target speaker output by the above model into the trained decoder to reconstruct the mel spectrum features of the converted sentence.

[0104] Step 7, use the Parallel WaveGAN vocoder to convert the mel spectrum features of the converted sentence obtained in step 6-2 into speech for output, thereby obtaining the synthesized speech of cross-lingual speech conversion.

[0105] The above describes exemplary embodiments of the present application, and does not limit the patent protection scope of the present application, any equivalent structure or equivalent process transformation using the content of the specification and drawings, or direct or indirect application in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A cross-language speech conversion method based on mutual information and SE attention mechanism, characterized in that, Includes the training phase and the transition phase; The training phase includes the following steps: Step 1: Obtain a large amount of training data, which consists of statements from multiple speakers in different languages; Step 2: The acquired training corpus is preprocessed and feature extracted. 128 frames are randomly selected from the Mel spectrum features of each training corpus and obtained by cascading operations to obtain Mel spectrum feature segments, which are represented as acoustic features X. Step 3: Input the acoustic features into the conversion network for training. The conversion network consists of four parts: content encoder, speaker encoder, pitch extractor, and decoder. The content encoder and speaker encoder are used to deentangle the speaker's speech to obtain the content information and speaker information of the sentence. The pitch extractor is used to extract the fundamental frequency F0 from the waveform of the speaker's speech. The decoder is used to reconstruct the obtained content information, speaker information, and fundamental frequency F0 to obtain the reconstructed Mel spectrum features, which are represented as acoustic features X′. Step 4: During the training of the transformation network, the reconstruction loss function of the transformation network is minimized. The hyperparameters of the transformation network are set to minimize the objective function until the set number of iterations is reached, thereby obtaining the trained transformation network. The conversion phase includes the following steps: Step 5: Select speech from different languages ​​as test data, and extract the source speaker's Mel spectral features X1 and the target speaker's Mel spectral features X2 from the test data as acoustic features. Step 6: Input the acoustic features X1 and X2 into the trained conversion network to reconstruct the Mel-spectral features X of the converted statement. 1→2 ; Step 7: Using a Parallel WaveGAN vocoder, the Mel spectrum features of the obtained converted sentences are converted into speech for output, thereby obtaining synthesized speech for cross-language speech conversion; In step 3, the content encoder consists of one h-net network, one quantizer q, and one g-net network. The h-net network consists of one one-dimensional standard convolutional layer with a stride of 2, four convolutional modules, and one 512-dimensional linear layer. There are four convolutional modules in total. The first three convolutional modules consist of layer normalization, a 512-dimensional linear layer, and a ReLU activation function. The last convolutional module consists of layer normalization, a 512-dimensional linear layer, an SE module, and a ReLU activation function. The SE module consists of one one-dimensional global average pooling layer, two fully connected layers, one ReLU activation function, one Sigmoid activation function, and dense residual connections.

2. The cross-language speech conversion method based on mutual information and SE attention mechanism according to claim 1, characterized in that, The training process in step 4 is as follows: 4.

1. The speaker's Mel spectral feature sequence X k ={x k,1 ,x k,2 ,…,x k,T The input to the h-net network of the content encoder yields the dense feature sequence Z. k ={z k,1 ,z k,2 ,…,z k,T / 2 The sequence length changes from T to T / 2; 4.

2. The dense feature sequence Z k ={z k,1 ,z k,2 ,…,z k,T / 2 The input is fed into the SE module, which consists of one one-dimensional standard average pooling layer, two fully connected layers, one ReLU activation function, one Sigmoid activation function, and dense residual connections. The two fully connected layers are alternated with the ReLU and Sigmoid activation functions. The SE module compresses local features into a single global context vector weight, and multiplies the global context vector weight with the feature elements to obtain the dense feature sequence Z. k Reweight and generate the output Z′ of the SE module k ={z′ k,1 ,z′ k,2 ,…,z′ k,T / 2 }; 4.3 The dense feature sequence Z′ after reweighting the SE module k ={z′ k,1 ,z′ k,2 ,…,z′ k,T / 2 The input is fed into quantizer q, which performs quantization based on the trainable codebook B, converting Z′ into its input. k ={z′ k,1 ,z′ k,2 ,…,z′ k,T / 2 Discretization yields dense feature sequences. It is the closest to Z′ in the trainable codebook B. k The vector, vector quantization removes Z′ by setting an information bottleneck. k Non-essential information in the text makes It is associated with the underlying speech information, and then the content encoder is trained by minimizing the loss function of the quantizer q; 4.

4. Dense feature sequences The input is fed into the g-net network to generate the aggregate sequence R. k ={r k,1 ,r k,2 ,…,r k,T / 2 }, given r k,t Contrastive predictive coding is used to distinguish positive samples after m steps by minimizing the loss function of the g-net network. And negative samples extracted from the negative sample set Ω, by predicting future samples with probabilistic contrastive loss, encode local features spanning multiple time steps into In the middle, As a representation of content; 4.

5. The speaker's Mel spectral feature sequence X k ={x k,1 ,x k,2 ,…,x k,T The input is fed into the speaker encoder to generate the speaker feature vector S. k , will S k As a representation of the speaker; 4.

6. The speaker's Mel spectral feature sequence X k ={x k,1 ,x k,2 ,…,x k,T The input is fed into a pitch extractor to extract the fundamental frequency F0, and F0 is logarithmically normalized to generate a speaker-independent fundamental frequency feature sequence P. k ={p k,1 ,p k,2 ,…,p k,T }; 4.7 Representing Content Speaker representation S k and fundamental frequency characteristic sequence P k ={p k,1 ,p k,2 ,…,p k,T The input is fed into the decoder for training, minimizing the decoder's loss function; 4.8 Representation of Calculation Content Speaker representation S k and fundamental frequency characteristic sequence P k The mutual information between each of the three factors is multiplied by a constant weight and added to the total loss of speech conversion, and then the total loss of speech conversion is minimized. 4.9 Repeat steps 4.1 to 4.8 until the set number of iterations is reached, thus obtaining the trained network.

3. The cross-language speech conversion method based on mutual information and SE attention mechanism according to claim 2, characterized in that, The input process in step 6 is as follows: 6.1 Input the Mel-spectral feature X1 of the source speaker into the content encoder and pitch extractor to generate content representations respectively. Fundamental frequency eigenvector P′ k The Mel spectrum feature X2 of the target speaker is input into the speaker encoder to generate the speaker representation S′. k ; 6.2 Representing Content Speaker representation S′ k and fundamental frequency eigenvector P′ k The input is fed into the decoder to generate the converted Mel-spectral features X. 1→2 .

4. The cross-language speech conversion method based on mutual information and SE attention mechanism according to claim 3, characterized in that, The total loss of the speech conversion is: L VC L VQ +L CPC +L REC +λ MI L MI Where, λ MI ≥0 represents the regularization parameter for mutual information loss, L VQ L CPC L REC L MI These represent the vector quantization loss of the quantizer q, the information noise contrast estimation loss of the contrastive predictive coding, the reconstruction loss of the decoder, and the mutual information loss, respectively.

5. The cross-language speech conversion method based on mutual information and SE attention mechanism according to claim 4, characterized in that, The loss function of the quantizer q is: Where K represents the number of training statements, T represents the number of fixed frames for the Mel-spectral features of each training statement, and z′ k,t This represents the dense feature sequence obtained after reweighting by the SE module. This represents the dense feature sequence after quantization of the trainable codebook, and sg(·) represents the stopping gradient operator. It represents the square of the 2-norm.

6. The cross-language speech conversion method based on mutual information and SE attention mechanism according to claim 5, characterized in that, The loss function for the contrastive predictive coding is: Where K represents the number of training statements, T represents the fixed number of frames for the Mel-spectral features of each training statement, M represents the step size for contrastive prediction, T′=T / 2-M, exp(·) represents e raised to the power of (·), W m Ω represents the trainable projection matrix. k,t,m It is a set of negative samples randomly selected from the current training statement. Ω represents the set of negative samples k,t,m Random samples in R k ={r k,1 ,r k,2 ,…,r k,T / 2 } represents a dense feature sequence The aggregated sequence output after passing through a 256-dimensional unidirectional recurrent neural network, r k,t Represents the aggregate sequence R k One sample.

7. The cross-language speech conversion method based on mutual information and SE attention mechanism according to claim 6, characterized in that, The reconstruction loss of the decoder is: Where K represents the number of training statements, T represents the number of fixed frames for the Mel-spectral features of each training statement, and x t This represents the input Mel-spectral feature segment. The spectral features reconstructed by the decoder are represented by ||·||1, which represents the 1-norm, and ||·||2, which represents the 2-norm.

8. The cross-language speech conversion method based on mutual information and SE attention mechanism according to claim 7, characterized in that, The mutual information loss is: in, These represent unbiased estimates of the upper bounds of the variational contrast logarithms between content representation and speaker representation, content representation and fundamental frequency feature, and fundamental frequency feature and speaker representation, respectively.

9. The cross-language speech conversion method based on mutual information and SE attention mechanism according to claim 8, characterized in that, The unbiased estimate of the upper bound of the variational contrast logarithm between the content representation and the speaker representation is: Where K represents the number of training statements, and T represents the number of fixed frames for the Mel-spectral features of each training statement. This represents the conditional log-likelihood of positive samples with respect to speaker representations and content representations. This represents the conditional log-likelihood of negative samples with respect to speaker representations and content representations. When representing a speaker's representation S, after... Content representation after network parameterization The corresponding variational approximation of the true posterior. This represents the content representation of the t-th frame corresponding to the k-th sentence in a positive sample pair. s represents the content representation of the t-th frame of the l-th sentence in the negative sample pair. k This represents the speaker representation of the k-th sentence in a positive sample pair. The unbiased estimate of the upper bound of the variational contrast logarithm between the content representation and the fundamental frequency feature is: Where K represents the number of training statements, and T represents the number of fixed frames for the Mel-spectral features of each training statement. This represents the conditional log-likelihood of positive samples with respect to fundamental frequency features and content representation. This represents the conditional log-likelihood of negative samples with respect to fundamental frequency features and content representation. This indicates that, given a fundamental frequency characteristic P, after... Content representation after network parameterization The corresponding variational approximation of the true posterior. This represents the content representation of the t-th frame corresponding to the k-th sentence in a positive sample pair. This represents the content representation of the t-th frame corresponding to the l-th sentence in the negative sample pair. The unbiased estimate of the upper bound of the variational contrast logarithm between the fundamental frequency feature and the speaker representation is: Where K represents the number of training statements, and T represents the number of fixed frames for the Mel-spectral features of each training statement. This represents the conditional log-likelihood of a positive sample with respect to speaker features and fundamental frequency features. This represents the conditional log-likelihood of negative samples with respect to speaker features and fundamental frequency features. When representing a speaker's representation S, after θ P,S The variational approximation of the true posterior corresponding to the network parameterized fundamental frequency feature P, p k,t Let s represent the fundamental frequency feature corresponding to the t-th frame of the k-th sentence in a positive sample pair. k p represents the speaker representation of the k-th sentence in a positive sample pair. l,t This represents the fundamental frequency feature corresponding to the t-th frame of the l-th sentence in the negative sample pair.

Images