Stylegan-based emotional speech conversion method based on fundamental frequency difference compensation

Abstract

The application discloses a StyleGAN (Style Generative Adversarial Network) emotional speech conversion method based on fundamental frequency difference compensation. First, the application proposes to take the multiple emotional style features extracted by a style encoder as label information, and adopts adaptive instance normalization to fuse the emotional style features with content features, so that the generator can not only fully learn the style features of the target emotion, but also realize the conversion between emotions not in the training set, that is, complete the conversion between any emotions in the open set situation. Further, on the basis of traditional logarithmic Gaussian normalized fundamental frequency conversion, the application proposes fundamental frequency difference compensation, thereby enhancing the amplitude difference between different emotions, solving the problem that after the traditional logarithmic Gaussian normalized fundamental frequency conversion, the fundamental frequency only presents an overall rise, and the mean and mean square deviation cannot accurately describe the fundamental frequency envelope amplitude difference of different emotions, effectively improving the emotional saturation of the converted speech, and realizing high-quality emotional speech conversion in the open set situation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of speech conversion, and particularly relates to a StyleGAN emotional speech conversion method based on fundamental frequency difference compensation. BACKGROUND

[0002] Speech conversion is an important branch of speech signal processing, and speech conversion aims to convert the individual characteristics of a source speaker into the individual characteristics of a target speaker while preserving the speech content of the source speaker. Speech conveys information through language and prosody, and prosody can not only affect the syntax and semantic interpretation of speech (language prosody), but also convey a person's emotional state (emotional prosody). Traditional speech conversion mainly focuses on the conversion of the individual characteristics of a speaker, and less on the conversion of the prosody of a speaker. Emotional speech conversion is to convert the emotional prosody of a source speaker into the emotional prosody of a target speaker while keeping the speech content and individual characteristics of the source speaker unchanged.

[0003] Early emotional speech conversion is obtained by training the spectral mapping between the source and target sentences. J. Tao et al. proposed to decompose the pitch contour of the source speech into a hierarchical structure using a classification regression tree, and then use a Gaussian Mixture Model (GMM) and a regression-based clustering method. Later, R. Ahara et al. proposed a sample-based conversion method, which encodes the source speech using parallel samples and then synthesizes the target speech. H. Ming et al. further extended this method to a unified sample-based emotional speech conversion framework, which simultaneously learns the mapping of spectral features and the fundamental frequency features based on the Continuous Wavelet Transform (CWT). With the advent of deep learning, the performance of speech conversion has been significantly improved. Methods based on Deep Neural Network (DNN), Deep Belief Network (DBN), Highway Network and Deep Bidirectional Long Short Term Memory Network (DBLASTM) are constantly proposed, which better realize the conversion of spectral features and prosodic features.

[0004] The emotional speech conversion method discussed above is basically based on conversion under parallel corpus, however, it is often difficult and time-consuming to collect a large amount of parallel emotional corpus; in addition, under the condition of parallel corpus, the emotional speech conversion method usually needs alignment operation in the training stage, which inevitably introduces additional distortion and affects the performance of the conversion model. Therefore, whether from the universality or practicability of the emotional speech conversion system, the emotional speech conversion research under the condition of non-parallel text has greater application value and practical significance. Therefore, people have proposed some methods to eliminate the demand of emotional speech conversion method for parallel data, such as the framework based on cycle consistent generative adversarial network (CycleGAN), based on variational auto-encoder (VAE), based on star generative adversarial network (StarGAN) and the like, but these methods are only focused on spectral feature conversion, and for fundamental frequency (F0), only the traditional log Gaussian normalization is used for conversion. Later, Kun Zhou et al. proposed a method based on CWT CycleGAN, CWT-based conditional variational auto-encoder and generative adversarial network (Variational Auto-encoder Wasserstein Generative Adversarial Network, VAWGAN), which inputs the F0 after CWT expansion into the model for training, but the conversion function still uses the traditional log Gaussian normalization. Later, Georgios Rizos et al. proposed an improved log Gaussian normalization F0 conversion function, which introduces the difference between the mean and mean square deviation of the source emotion and the target emotion on the basis of the traditional log Gaussian normalization, but when the amplitude of the emotion changes too much, the mean and mean square deviation cannot accurately describe the specific fluctuation difference of the fundamental frequency envelope amplitude of the two emotions, and only the amplitude of the source emotion can be processed to the amplitude range of the target emotion.

[0005] Further, most of the above-mentioned methods are emotional speech conversion under closed set, that is, the corpus of target emotion participates in the training, and the training data is labeled, in this case, the quality of the converted speech is relatively good, however, in actual application scenarios, the target emotion may only have a few corpora or only one sentence participating in the training, or even no training, such problems can be divided into emotional speech conversion under open set, that is, arbitrary emotional speech conversion problem. For how to improve the sound quality and emotional saturation of emotional conversion speech under open set, it has become a research hotspot and difficulty in the field. SUMMARY

[0006] To solve the above technical problems, the application provides a StyleGAN emotional speech conversion method based on fundamental frequency difference compensation, which uses a style encoder to extract emotional style features as labels for training, realizes emotional speech conversion in an open set, and solves the problems of existing methods that require labeled training data and are only applicable to closed sets. Furthermore, a fundamental frequency difference compensation vector is proposed to solve the problem that after traditional log Gaussian normalization is used for fundamental frequency conversion, the converted fundamental frequency only shows an overall rise, and the mean and variance cannot accurately describe the differences between the amplitudes of different emotions, thereby improving the emotional saturation of the converted speech and realizing high-quality emotional speech conversion in an open set.

[0007] To solve the above technical problems, the application adopts the following technical solutions:

[0008] The application first proposes a StyleGAN emotional speech conversion method based on fundamental frequency difference compensation, which includes a training phase and a conversion phase. The training phase includes the following steps:

[0009] Step 1: Obtain training corpus, which is composed of multiple emotional corpora of one speaker, including source emotions and target emotions;

[0010] Step 2: Extract the spectral features of different emotional speech from the training corpus as acoustic feature vectors;

[0011] Step 3: Input the obtained acoustic feature vectors into the StyleGAN-EVC network for training, and continuously optimize the objective function until the set number of iterations is reached, thereby obtaining a trained StyleGAN-EVC network; the StyleGAN-EVC network includes a generator G, a discriminator D, and a style encoder S;

[0012] The generator G is divided into an encoder and a decoder, the encoder is used to generate content features, and the decoder is used to reconstruct the obtained content features and the style features extracted by the style encoder S to generate reconstructed speech;

[0013] Step 4: Construct a fundamental frequency conversion function from the source emotion to the target emotion, and on the basis of the traditional log Gaussian normalized fundamental frequency conversion, introduce a fundamental frequency difference compensation vector to construct the final fundamental frequency conversion function;

[0014] The conversion phase includes the following steps:

[0015] Step 5: Select different emotional speech of one speaker as the to-be-converted corpus, and extract the source emotion mel-frequency spectrum features x s and the target emotion mel-frequency spectrum features xt , corresponding log fundamental frequency feature logf 0s and logf 0t , and corresponding aperiodic feature AP s and AP t as an acoustic feature vector;

[0016] Step 6, input the source emotional spectrum feature x s and the target emotional spectrum feature x t into the trained StyleGAN-EVC network of step 3 to reconstruct the converted emotional spectrum feature x st ;

[0017] Step 7, convert the source emotional log fundamental frequency feature logf 0s extracted in step 5 into the target emotional fundamental frequency feature f 0st by using the fundamental frequency conversion function obtained in step 4;

[0018] Step 9, synthesize the converted emotional speech by using the WORLD vocoder with the converted emotional spectrum feature x st generated in step 6, the converted emotional fundamental frequency feature f 0st obtained in step 7, and the source emotional aperiodic feature AP s extracted in step 5.

[0019] Further, the style encoder S is composed of 5 one-dimensional pooling modules and 5 one-dimensional convolution modules, wherein each one-dimensional pooling module is composed of an average pooling, each one-dimensional convolution module includes convolution and ReLU activation function, and the output layer is composed of a full connection layer.

[0020] Further, the training process of step 3 includes the following steps:

[0021] Step 3-1, input the source emotional spectrum feature x s into the encoding network of the generator G to obtain the emotion-independent semantic feature G(x s );

[0022] Step 3-2, input the target emotional spectrum feature x t into the style encoder S to obtain the target emotional style feature s t ;

[0023] Step 3-3, input the generated semantic feature G(x s ) and the target emotional style feature s t into the decoding network of the generator G for training, and minimize the loss function of the generator G in the training process, so as to obtain the converted emotional spectrum feature x st ;

[0024] Step 3-4: The spectrum feature x of the source emotion s Input style encoder S to obtain the style feature s of the source emotion s ;

[0025] Step 3-5: Convert the above generated spectrum features x of the converted emotion st Input the encoding network of the generator G again to obtain the emotion-independent semantic feature G(x st );

[0026] Step 3-6: Generate the semantic features G(x st ) and the style characteristics of the source emotion s s , input the decoding network of generator G for training, minimize the loss function of generator G during the training process, and obtain the spectral characteristics of the reconstructed source emotion

[0027] Step 3-7: Convert the spectral feature x of the converted emotion generated in step 3-3 st Input into the discriminator D for training, minimizing the loss function of the discriminator D;

[0028] Step 3-8: Convert the spectral feature x of the converted emotion generated in step 3-3 st , input style encoder S for training, and minimize the style loss function of style encoder S;

[0029] Step 3-9: Return to step 3-1 and repeat the above steps until the set number of iterations is reached, thereby obtaining the trained StyleGAN-EVC network.

[0030] Furthermore, the conversion process of step 6 includes the following steps:

[0031] Step 6-1: The spectrum feature x of the source emotion s Input the encoding network of the generator G trained in step 3 to obtain the emotion-independent semantic features G(x s );

[0032] Step 6-2: The spectrum feature x of the target emotion t Input the style encoder S to obtain the style feature s of the target emotion t ;

[0033] Step 6-3: The above generated emotion-independent semantic features G(x s ) and the style features of the target emotion s t Input the decoder network of the generator G trained in step 3 to obtain the spectrum feature x of the converted emotion st .

[0034] Further, the style reconstruction loss function of the style encoder S is represented as:

[0035]

[0036] wherein, represents the expectation of the difference between the style feature of the target emotion generated by the style encoder and the style feature of the converted emotion, ||·||1 represents the 1-norm, S(·) is the style encoder, S(x t ) represents the style feature of the target emotion generated by the style encoder, G(·) is the generator, G(x s ,S(x t )) represents the spectrum feature of the converted emotion generated by the generator, S(G(x s ,S(x t )) represents the style feature of the converted emotion generated by the style encoder, x s is the spectrum feature of the source emotion, and x t is the spectrum feature of the target emotion.

[0037] Further, the objective function of the StyleGAN-EVC network is represented as:

[0038] L StyleGAN =L G +L D ,

[0039] wherein, L G is the loss function of the generator, and L D is the loss function of the discriminator;

[0040] The loss function L G of the generator is represented as:

[0041]

[0042] wherein, λ cyc and λ sty are a set of regularization hyperparameters, respectively representing the weights of the cycle consistency loss and the style reconstruction loss, and respectively represent the adversarial loss of the generator, the cycle consistency loss, and the style reconstruction loss of the style encoder;

[0043] The loss function L D of the discriminator is:

[0044]

[0045] wherein, is the adversarial loss of the discriminator.

[0046] Further, the fundamental frequency conversion function is:

[0047]

[0048] Wherein, mu s And mu t Respectively represent the mean of the logarithmic fundamental frequency characteristics of the source emotion and the target emotion, sigma s And sigma t Respectively represent the mean square deviation of the logarithmic fundamental frequency characteristics of the source emotion and the target emotion, theta represents the fundamental frequency difference compensation vector.

[0049] The fundamental frequency difference compensation vector theta is expressed as:

[0050]

[0051] Wherein, The fundamental frequency characteristics of the target emotion are obtained by linear interpolation or uniform sampling, mu t ' represents the mean of the fundamental frequency characteristics of the target emotion.

[0052] The beneficial effects of the present application are: (1) the present application extracts the emotional style features of the style encoder as label information for training, compared with the traditional one-hot vector which only has an indication function and carries too little specific emotional information, it is more conducive to the decoding network to learn more emotional style information, improve the sound quality and emotional saturation of the converted speech, and obtain better emotional conversion speech;

[0053] (2) the present application proposes a fundamental frequency difference compensation vector, which corrects the amplitude change of emotion and improves the emotional saturation of the converted speech, overcoming the problem that the traditional logarithmic Gaussian normalized fundamental frequency function conversion only rises as a whole, and the mean and mean square deviation cannot accurately describe the difference in amplitude of different emotions;

[0054] (3) the training network of the present application is more efficient and stable, and can complete the emotional speech conversion under the open set condition, that is, the target emotion is not involved in the training, and the emotional speech conversion is realized, and only a small amount of corpus of target sentences is required during conversion, solving the problem that the target emotion needs to participate in the training in practical application. Therefore, the present application is an emotional speech conversion method with high sound quality and emotional saturation under the open set condition. BRIEF DESCRIPTION OF DRAWINGS

[0055] Figure 1 is a schematic diagram of the model according to the embodiment of the present application;

[0056] Figure 2 is a network structure diagram of the style encoder of the model according to the embodiment of the present application;

[0057] Figure 3is a network structure diagram of a generator of the model described in the embodiment of the present application;

[0058] Figure 4 is a network structure diagram of a discriminator of the model described in the method embodiment;

[0059] Figure 5 is a schematic diagram of the fundamental frequency conversion principle of the model described in the embodiment of the present application. DETAILED DESCRIPTION

[0060] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0061] The present application proposes a StyleGAN emotional voice conversion method based on fundamental frequency difference compensation. The StyleGAN model is applied to emotional voice conversion, and the emotional style features extracted by the style encoder are used as labels and input into the decoder part of the generator together with the content features separated by the encoder of the generator for reconstruction, so as to generate converted emotional voice, overcoming the problem that the training data in the traditional model needs to have labels. Secondly, in the fundamental frequency conversion part, on the basis of the traditional log Gaussian normalized fundamental frequency conversion, a fundamental frequency difference compensation vector is introduced to enhance the amplitude change of emotion, so as to generate converted voice with higher emotional saturation. The converted emotional voice after the traditional log Gaussian normalized fundamental frequency function conversion only presents an overall pitch rise or fall, but the amplitude change trend is similar and has no distinguishing degree. Further, the present method can realize emotional voice conversion in an open set, i.e. emotional conversion under the condition that the target emotion does not appear in the training set, and in the conversion stage, the target emotion only needs a small amount of corpus to realize conversion, which is more suitable for practical application. The proposed emotional voice conversion based on StyleGAN is called StyleGAN-EVC (Emotional Voice Conversion With StyleGAN).

[0062] As shown in Figure 1 , the method of the present embodiment is divided into two parts: a training part and a conversion part. The former is used to obtain a model for conversion, and the latter is used to realize conversion of the source emotion to the target emotion.

[0063] The training stage includes the following steps:

[0064] Step 1, obtain training corpus, the training corpus comes from ESD emotion corpus, the corpus has 10 Chinese and 10 English speakers, each language has 5 male and 5 female speakers, each speaker has 5 kinds of emotional sentences, respectively neutral, angry, happy, sad and surprised, each person has 350 sentences for each emotion, 281 sentences are selected as training corpus, 69 sentences are selected as test corpus. In this experiment, the emotional corpus of a female English speaker is selected, and the training corpus of neutral, angry and sad emotions is selected as the training set, and the test corpus of neutral, angry, happy, sad and surprised emotions is selected as the test set.

[0065] Step 2, the training corpus extracts the spectral envelope feature x, the aperiodicity feature AP and the logf0 of each emotional sentence through the WORLDCoder.

[0066] Step 3, the StyleGAN-EVC network in this embodiment is improved on the basis of the StyleGAN model and applied to emotional speech conversion, on the one hand, the emotional style features extracted by the style encoder are input as labels into the decoding part of the generator, and instance normalization is used to fully fuse the content features extracted by the encoding part of the generator, and further, on the basis of the traditional log Gaussian normalization fundamental frequency conversion function, a fundamental frequency difference compensation vector is introduced, and the amplitude change degree of emotion is enhanced. The StyleGAN-EVC network in this embodiment is composed of three parts: generator G, discriminator D and style encoder S.

[0067] The objective function of the StyleGAN-EVC network in this embodiment is represented as:

[0068] L StyleGAN =L G +L D ,

[0069] Wherein, L G is the loss function of the generator, L D is the loss function of the discriminator;

[0070] The loss function L G of the generator is represented as:

[0071]

[0072] Wherein, λ cyc and λ sty are a set of regularization hyperparameters, representing the weights of the cycle consistency loss and the style reconstruction loss respectively, and represent the adversarial loss of the generator, the cycle consistency loss, and the style reconstruction loss of the style encoder respectively;

[0073] The loss function L of the discriminator D for:

[0074]

[0075] in, is the adversarial loss of the discriminator.

[0076] Step 4: The spectral feature x of the target emotion t Input the style encoder S to obtain the style feature s of the target emotion t .

[0077] like Figure 2 As shown in the figure, the style encoder uses a 1D convolutional neural network with a ReLU activation function. The style encoder consists of five layers of 1D convolutional modules and five layers of 1D pooling modules. Each layer of the 1D convolutional module includes convolution and ReLU activation functions, and each layer of the 1D pooling module consists of average pooling. The output layer consists of a fully connected layer.

[0078] Step 5: Extract the spectrum feature x of the source emotion s The style feature s of the target emotion obtained in step 4 t Input the generator together for training, so that the generator's loss function L G As small as possible, get the spectrum feature x of the generated conversion emotion st .

[0079] like Figure 3 As shown in the figure, the generator uses a 2D dynamic convolutional network with a Mish function as the activation function. The generator consists of an encoder and a decoder. The encoder network consists of seven 2D convolutional modules. The first three layers are 2D convolutional modules, each of which includes 2D convolution, instance normalization, and a Mish function. The last four layers are dynamic convolutional modules, each of which includes dynamic convolution, instance normalization, and a Mish function. The decoder network consists of six 2D convolutional modules. The first four layers are dynamic convolutional modules, each of which includes dynamic convolution, adaptive instance normalization, and a Mish function. The last two layers are 2D transposed convolutional modules, each of which includes transposed dynamic convolution, adaptive instance normalization, and a Mish function.

[0080] Step 6: The spectrum feature x of the converted emotion generated in step 5 st And the spectrum feature x of the target emotion obtained in step 2 t Input the discriminator, train the discriminator, and make the discriminator's adversarial loss function As small as possible.

[0081] like Figure 4As shown, the discriminator is composed of 5 layers of 2-dimensional convolution modules and an output layer. Each layer of 2-dimensional convolution module includes 2-dimensional convolution and LeakyReLU function, and the output layer of the discriminator is set to 1 in the convolution channel number.

[0082] The loss function of the discriminator is:

[0083]

[0084] wherein, is the adversarial loss of the discriminator.

[0085]

[0086] wherein, D(x s ) represents that the discriminator D discriminates the real spectrum feature, s t represents the style feature of the target emotion generated by the style encoder S, i.e., S(x t )=s t , G(x s , s t ) represents the converted emotion spectrum feature generated by the generator G, D(G(x s , s t )) represents that the discriminator discriminates the generated spectrum feature, represents the expectation of the probability distribution generated by the generator G, represents the expectation of the real probability distribution;

[0087] The optimization objective is:

[0088]

[0089] Step 7, input the spectrum feature x st of the target emotion obtained in step 5 into the encoding network of the generator G again to obtain the emotion-independent semantic feature G(x st ), input the spectrum feature x s of the source emotion into the style encoder S to obtain the style feature s s of the source emotion, and input the obtained semantic feature G(x st ) and the style feature s s of the source emotion into the decoding network of the generator G for training, wherein the loss function of the generator G is minimized in the training process, and the reconstructed spectrum feature x s of the source emotion is obtained.

[0090] The loss function of the generator is minimized in the training process, including the adversarial loss of the generator, the cycle consistency loss, and the style reconstruction loss of the style encoder. The training cycle consistency loss is to make the reconstructed spectrum feature x s of the source emotion after passing through the generator G can be and x s To keep consistent as much as possible, the style reconstruction loss is to constrain the style encoder to generate more style features s t ;

[0091] The loss function of the generator is:

[0092]

[0093] The optimization objective is:

[0094]

[0095] where λ cyc and λ sty are a set of regularization hyperparameters, representing the weights of the cycle-consistency loss and the style reconstruction loss, respectively.

[0096] The adversarial loss of the generator in GAN is:

[0097]

[0098] where E represents the expectation of the probability distribution generated by the generator, s t represents the style feature of the target emotion generated by the style encoder, i.e. S(x t ) = s t , G(x s , s t ) represents the converted emotion spectrum feature generated by the generator, and D(G(x s , s t )) represents the discriminator's judgment of the real target spectrum feature. The loss of the discriminator D together with the loss of the generator G constitute the common adversarial loss in GAN, which is used to distinguish whether the input spectrum to the discriminator is real or generated. In the training process s as small as possible, the generator is constantly optimized until it generates a spectrum feature G(x s , s s ) that is difficult for the discriminator to distinguish between real and fake.

[0099] The cycle-consistency loss in the generator G is:

[0100]

[0101] where s s represents the style feature of the source emotion, i.e. S(x s ) = s ss t ),s s ) is the reconstructed source emotion spectrum feature generated by the generator, is the loss expectation of the reconstructed source emotion spectrum and the real source emotion spectrum, and ||·||1 represents the 1-norm. In the loss of training the generator, is as small as possible, so that the spectrum feature G(x s ,s t ) of the target emotion generated by the generator is as similar as possible to x s . After being input into the generator again, the obtained spectrum feature of the reconstructed source emotion speech is as similar as possible to x s . By training , it can be effectively ensured that the semantic feature of the emotion speech is not lost after being encoded by the generator.

[0102] is the style reconstruction loss of the style encoder S, which is used to optimize the style feature s t :

[0103]

[0104] wherein s t represents the style feature of the target emotion generated by the style encoder S, that is, S(x t ) = s t , G(x s ,s t ) represents the spectrum feature of the converted emotion generated by the generator, ||·||1 represents the 1-norm, and S(G(x s ,s t )) represents the style feature of the converted emotion generated by the style encoder S.

[0105] The spectrum feature G(x s ,s t ) of the target emotion is input into the style encoder S to obtain the reconstructed style feature, and the absolute value of the difference between the reconstructed style feature and the style feature s t of the target emotion generated by the style encoder is calculated, and in the training process, is as small as possible, so that the style feature s t of the target emotion generated by the style encoder S can fully express the features of the target emotion.

[0106] Step 8, repeat steps 4 to 7 until the set number of iterations is reached, so as to obtain the trained StyleGAN-EVC network. Due to different specific settings of neural networks and different performance of experimental equipment, the set number of iterations is also different. In this experiment, the number of iterations is set to 200,000 times.

[0107] Step 9, using log Gaussian normalized fundamental frequency conversion, the mean and mean square deviation of the log fundamental frequency of each emotion are counted, and the log fundamental frequency feature of the source emotion is converted to the log fundamental frequency feature of the target emotion . The mean of the fundamental frequency feature of the target emotion is used to obtain the fundamental frequency difference compensation vector. On the basis of the traditional log Gaussian normalized fundamental frequency conversion function, the fundamental frequency difference compensation vector is introduced to obtain the final fundamental frequency conversion function.

[0108] As shown in Figure 5 , the fundamental frequency conversion function is:

[0109]

[0110] wherein, is the log fundamental frequency feature of the converted emotion after log Gaussian normalized fundamental frequency conversion, and θ is the fundamental frequency difference compensation vector.

[0111]

[0112]

[0113] wherein, μ s and μ t represent the mean of the log fundamental frequency feature of the source emotion and the target emotion respectively, σ s and σ t represent the mean square deviation of the log fundamental frequency feature of the source emotion and the target emotion respectively. Since the selected sentences of the source emotion and the target emotion are non-parallel corpus, the fundamental frequency features of the target emotion and the source emotion are not aligned. If the dimension of the fundamental frequency feature of the source emotion is larger than that of the target emotion, uniform sampling is adopted, otherwise linear interpolation is performed to obtain the aligned fundamental frequency feature of the target emotion and the source emotion, i.e. μ t ' represents the mean of the fundamental frequency feature of the target emotion.

[0114] The conversion stage includes the following steps:

[0115] Step 10, using the WORLDEncoder to extract the spectral feature, the non-periodic feature and the log fundamental frequency feature of the source emotion and the target emotion.

[0116] Step 11, inputting the spectral feature of the target emotion extracted in step 10 into the style encoder S to obtain the style feature s t of the target emotion.

[0117] Step 12, inputting the spectral feature x s of the source emotion obtained in step 10 and the style feature of the target emotion obtained in step 10 into the trained StyleGAN-EVC network to reconstruct the spectral feature xst .

[0118] Step 13, the fundamental frequency conversion function obtained by step 9 is used to convert the logarithmic fundamental frequency features of the source emotion extracted in step 10 into the fundamental frequency features of the target emotion.

[0119] Step 14, the WORLD encoder is used to synthesize the converted target emotion speech by using the spectral features x st of the target emotion obtained in step 12, the converted emotion fundamental frequency features obtained in step 13, and the non-periodic features extracted in step 10.

[0120] The above describes the exemplary embodiments of the present application, and does not limit the patent protection scope of the present application. Any equivalent structure or equivalent flow transformation using the content of the specification and the drawings, or direct or indirect application in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A StyleGAN emotion speech conversion method based on fundamental frequency difference compensation, characterized in that, The method comprises a training phase and a conversion phase, and the training phase comprises the following steps: Step 1, obtaining a training corpus, the training corpus is composed of multiple emotion corpora of a speaker, and the emotions include source emotions and target emotions; Step 2, extracting the spectral features of different emotional speech as acoustic feature vectors from the training corpus; Step 3, inputting the obtained acoustic feature vectors into a StyleGAN-EVC network for training, constantly optimizing the objective function until the set number of iterations is reached, thereby obtaining a trained StyleGAN-EVC network; the StyleGAN-EVC network comprises a generator G, a discriminator D and a style encoder S; The generator G is divided into an encoder and a decoder, the encoder is used to generate content features, and the decoder is used to reconstruct the obtained content features and the style features extracted by the style encoder S to generate reconstructed speech; Step 4, constructing a fundamental frequency conversion function from the source emotion to the target emotion, introducing a fundamental frequency difference compensation vector on the basis of the traditional log Gaussian normalized fundamental frequency conversion to construct a final fundamental frequency conversion function; The conversion phase comprises the following steps: Step 5, select different emotional speech of a speaker as the speech to be converted, and extract the source emotional mel-frequency spectrum feature x s and the target emotional mel-frequency spectrum feature x t , the corresponding log fundamental frequency feature and and the corresponding non-periodic feature AP s and AP t as the acoustic feature vector; Step 6, input the source emotion spectrum feature x s and the target emotion spectrum feature x t into the trained StyleGAN-EVC network in step 3 to reconstruct the converted emotion spectrum feature x st ; Step 7, converting the source emotion log fundamental frequency features extracted in step 5 into target emotion log fundamental frequency features by the fundamental frequency conversion function obtained in step 4 Step 7, converting the source emotion log fundamental frequency features extracted in step 5 into target emotion log fundamental frequency features by the fundamental frequency conversion function obtained in step 4 Step 9, the converted emotion spectrum feature x st , the converted emotion fundamental frequency feature obtained in Step 7 The source emotion aperiodic feature AP extracted in Step 5 s The converted emotion speech is synthesized by a WORLd vocoder.

2. The style-based generator (StyleGAN) emotion speech conversion method based on fundamental frequency difference compensation according to claim 1, characterized in that, The style encoder S is composed of 5 one-dimensional pooling modules and 5 one-dimensional convolution modules, wherein each one-dimensional pooling module is composed of an average pooling, each one-dimensional convolution module comprises convolution and a ReLU activation function, and the output layer is composed of a full connection layer.

3. The method of claim 1, wherein the method is based on a fundamental frequency difference compensation-based StyleGAN emotional speech conversion method. The training process of step 3 comprises the following steps: Step 3-1, obtaining the spectral feature x of the source emotion s In the encoding network of the input generator G, the emotion-independent semantic feature G(x s ) is obtained. Step 3-2, obtaining the spectral feature x of the target emotion t inputting the style encoder S to obtain the style feature s of the target emotion t ; Step 3-3, input the generated semantic feature G(x s ) and the style feature s t of the target emotion into the decoding network of the generator G, train in the process, minimize the loss function of the generator G, so as to obtain the spectrum feature x st of the converted emotion; Step 3-4, obtain the spectral feature x of the source emotion s Input the style encoder S to obtain the style feature s of the source emotion s ; Step 3-5, the generated conversion emotion spectrum feature x st The encoding network of the generator G is input again to obtain the emotion-independent semantic feature G(x st ); Step 3-6, the generated semantic feature G(x st ) above and the style feature s s of the source emotion are input into the decoding network of the generator G, and in the training process, the loss function of the generator G is minimized to obtain the reconstructed spectral feature of the source emotion Step 3-7, the generated converted emotion spectrum feature x in step 3-3 st The input discriminator D is trained to minimize the loss function of the discriminator D. Step 3-8, the converted emotion spectrum feature x generated in step 3-3 is input into the style encoder S for training, so as to minimize the style loss function of the style encoder S. st , inputting the style encoder S for training, minimizing the style loss function of the style encoder S. Step 3-9, returning to step 3-1 to repeat the above steps until the set number of iterations is reached, thereby obtaining a trained StyleGAN-EVC network.

4. The method of claim 1, wherein the method is based on a fundamental frequency difference compensation-based StyleGAN emotional speech conversion method. The conversion process of step 6 comprises the following steps: Step 6-1, obtaining the spectral feature x of the source emotion s Input the encoding network of the trained generator G in step 3 to obtain the emotion-independent semantic feature G(x s ); Step 6-2, obtaining the spectral feature x of the target emotion t In the input style encoder S, obtaining the style feature s of the target emotion t ; Step 6-3, input the generated emotion-independent semantic features G(x s ) and the style features s t of the target emotion into the decoding network of the trained generator G to obtain the spectrum features x st of the converted emotion.

5. The method of claim 1, wherein the method is based on a fundamental frequency difference compensation-based StyleGAN emotional speech conversion method. The style reconstruction loss function of the style encoder S is represented as: wherein, denotes the expectation of the difference between the style feature of the target emotion generated by the style encoder and the style feature of the converted emotion, ||·||1 denotes the 1-norm, S(·) is the style encoder, S(x t ) denotes the style feature of the target emotion generated by the style encoder, G(·) is the generator, G(x s ,S(x t )) denotes the spectrum feature of the converted emotion generated by the generator, S(G(x s ,S(x t )) denotes the style feature of the converted emotion generated by the style encoder, x s is the spectrum feature of the source emotion, x t is the spectrum feature of the target emotion.

6. The method of claim 1, wherein the method is based on a fundamental frequency difference compensation-based StyleGAN emotional speech conversion method. The objective function of the StyleGAN-EVC network is represented as: L StyleGAN = L G + L D , where L G is the loss function for the generator, L D is the loss function for the discriminator; Loss function L of the generator G is represented as: where λ cyc and λ sty are a set of regularization hyperparameters, representing the weights of the cycle-consistency loss and the style reconstruction loss, respectively, and represent the adversarial loss, the cycle-consistency loss, and the style reconstruction loss of the style encoder, respectively. Loss function L of discriminator D is: wherein, is the discriminator's adversarial loss.

7. The method of claim 1, wherein the method is based on a fundamental frequency difference compensation-based StyleGAN emotional speech conversion method. The fundamental frequency conversion function is: wherein μ s and μ t respectively represent the mean of the log fundamental frequency features of the source emotion and the target emotion, σ s and σ t respectively represent the mean square deviation of the log fundamental frequency features of the source emotion and the target emotion, and θ represents the fundamental frequency difference compensation vector; The fundamental frequency difference compensation vector θ is represented as: wherein, the fundamental frequency features of the target emotion obtained by linear interpolation or uniform sampling, μ t ′ the mean of the fundamental frequency features of the target emotion.

Images