Voice generation method for non-parallel voice conversion

Abstract

The invention discloses a voice generation method for non-parallel voice conversion, which relates to the technical field of intelligent voice processing, and comprises the following steps: respectively adding self-attention mechanisms between multiple residual error layers of a CycleGAN-VC2 network model generator and after a downsampling first layer of a discriminator; the improved CycleGAN-VC2 network model can better pay attention to the authenticity of voice local features and more accurately distinguish weighted features, so that the similarity of voice conversion is improved, and the phenomenon that the voice features in non-parallel corpora greatly fluctuate in a time period is weakened; the improved CycleGAN-VC2 network model is trained by using the target voice MCEP features and the standard voice data set, so that the trained CycleGAN-VC2 network model is more biased to the conversion of the target voice, and then the standard voice of the text of the voice needing to be converted is input into the trained CycleGAN-VC2 network model and is processed to obtain the required voice. Therefore, the similarity between the converted voice and the target voice is greatly improved, and voice generation of any speaker is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent speech processing, and in particular to a speech generation method for non-parallel speech conversion. Background Art

[0002] The purpose of speech conversion is to change the voice of the source speaker by processing acoustic features such as spectrum and rhythm without changing the content of the speech, so that the changed voice sounds like the target speaker; text-to-speech is one of the key technologies of human-computer interaction and has important research value and application prospects in the field of artificial intelligence; combining speech conversion and speech synthesis technology can realize a more intelligent and personalized speech generator.

[0003] Speech generator technology has been widely used in intelligent navigation, short video dubbing, etc.; according to whether the speech content of the training speech data set is consistent, speech conversion is divided into parallel speech conversion and non-parallel speech conversion. Parallel speech conversion has good effect, but parallel corpus is difficult to collect and requires preprocessing operations such as time alignment, otherwise the conversion effect will not be ideal; in contrast, non-parallel speech conversion does not require the use of parallel corpus and does not require time alignment, thus reducing the cost of data set production. Therefore, non-parallel speech conversion is currently one of the hot research directions. The non-parallel speech conversion currently used generally uses non-parallel corpus, decouples the acoustic features into linguistic features and speaker representations, and then maps the speech features from the source speaker to the target speaker to achieve conversion.

[0004] However, the speech features in non-parallel corpora fluctuate greatly within a period of time, resulting in a low similarity between the converted speech and the target speech, making it difficult to achieve accurate speech conversion. Summary of the invention

[0005] The embodiment of the present invention provides a speech generation method for non-parallel speech conversion, which can solve the problem in the prior art that the speech features in the non-parallel corpus fluctuate greatly within a time period, resulting in a low similarity between the converted speech and the target speech.

[0006] The embodiment of the present invention provides a speech generation method for non-parallel speech conversion, comprising the following steps:

[0007] Obtain the target's speech recording data and form the target's speech data set; at the same time obtain the standard sound data set;

[0008] Constructing a CycleGAN-VC2 network model, respectively adding a first self-attention mechanism between the multi-layer residual modules of the CycleGAN-VC2 network model generator to form an improved generator; adding a second self-attention mechanism after the downsampling first layer of the CycleGAN-VC2 network model discriminator to form an improved discriminator; and forming an improved CycleGAN-VC2 network model;

[0009] Extract the target speech MCEP features in the target speech data set and the standard speech MCEP features in the standard speech data set; use the target speech MCEP features and the standard speech MCEP features to train the improved CycleGAN-VC2 network model, and during the training, first fix the generator and train the discriminator, and at the same time fix the discriminator and train the generator to obtain the trained CycleGAN-VC2 network model;

[0010] Convert the text to be generated into standard pronunciation, extract the MCEP features of the standard pronunciation, and input the MCEP features of the standard pronunciation into the generator of the trained CycleGAN-VC2 network model to obtain the converted text MCEP features;

[0011] The fundamental frequency of the standard sound of the text is extracted and converted, and the non-periodic features of the target speech data set are extracted; the required speech is synthesized by combining the converted text MCEP features, the converted fundamental frequency of the standard sound of the text and the non-periodic features of the target speech data set.

[0012] Preferably, the acquisition of the standard sound data set includes:

[0013] Using news and prose text as materials, and using pyttsx3 to generate male and female standard voice datasets.

[0014] Preferably, the improved generator comprises:

[0015] The improved generator is a cascade of downsampling layers, residual layers, self-attention mechanisms, and upsampling layers;

[0016] The downsampling layer is 2D to 1D, and the upsampling layer is 1D to 2D;

[0017] The self-attention mechanism is deployed between the 6 residual layers.

[0018] Preferably, the improved discriminator comprises:

[0019] The improved discriminator is a cascade of 1D CNN, 2D CNN, GLU layer and Instance normalization layer;

[0020] The 1D CNN, Instance normalization layer and GLU layer in the discriminator constitute the downsampling layer of the discriminator;

[0021] A self-attention mechanism is deployed after the first layer of the downsampling layer.

[0022] Preferably, the process of extracting features by the self-attention mechanism includes:

[0023] The self-attention mechanism extracts the input query Query, input key Key, and input value Value information through three different convolutional layers;

[0024] Since the weight parameters of the three convolutional layers are different, when extracting information from the same data, the extracted feature vectors will be different. The extracted feature vectors are obtained using the self-attention mechanism to generate a weighted summation result. The way the self-attention mechanism extracts features is as follows:

[0025]

[0026] Where: Q is the input query; K is the input key; V is the input value; d k The input dimensions of Query and Key; QK T Score the similarity between the query and the key.

[0027] Preferably, the improved CycleGAN-VC2 network model comprises:

[0028] The improved CycleGAN-VC2 network model includes 2 generators and 4 discriminators;

[0029] The 2 generators are: G X→Y and G Y→X ; The 4 discriminators are: D X , D Y and D′ for generating the quadratic adversarial loss X and D′ Y ;

[0030] The total loss function of the improved CycleGAN-VC2 network model consists of primary adversarial loss, secondary adversarial loss, cycle consistency loss and identity mapping loss, and its formula is:

[0031] L=L adv +L adv2 +λ cyc L cyc (G X→Y , G Y→X )+λ id L id (G X→Y , G Y→X)

[0032] Where: L is the total loss function; L adv For a loss against; L adv2 is the secondary confrontation loss; L cyc (G X→Y , G Y→X ) is the cycle consistency loss; λ cyc is the hyperparameter of cycle consistency loss; L id (G X→Y , G Y→X ) is the identity mapping loss; λ id Hyperparameters for the identity mapping loss.

[0033] Preferably, the extracting and converting the fundamental frequency of the standard pronunciation of the text includes:

[0034] Use PyWORLD to extract the logarithmic fundamental frequency mean, variance, non-periodic features and MCEP features of the target speech dataset and the standard sound dataset;

[0035] PyWORLD is used to extract the fundamental frequency f0 of the standard text sound. According to the logarithmic fundamental frequency mean and variance of the target speech dataset and the standard sound dataset, the fundamental frequency f0 of the standard text sound is converted to obtain the converted fundamental frequency f0'. The conversion method is:

[0036]

[0037] Where: f′0 is the converted fundamental frequency; μ′ and σ′ are the mean and mean square error of the converted fundamental frequency in the logarithmic domain; μ and σ are the mean and mean square error of the source speech fundamental frequency in the logarithmic domain; f0 is the source speech fundamental frequency.

[0038] The embodiment of the present invention provides a speech generation method for non-parallel speech conversion, which has the following beneficial effects compared with the prior art:

[0039] The present invention adds self-attention mechanisms between the multi-layer residual layers of the CycleGAN-VC2 network model generator and after the first downsampling layer of the discriminator, so that the improved CycleGAN-VC2 network model can better focus on local speech features and more accurately distinguish the authenticity of weighted features, so as to improve the similarity of speech conversion and reduce the phenomenon that the speech features in non-parallel corpora fluctuate greatly within a time period.

[0040] At the same time, the improved CycleGAN-VC2 network model is trained using the target speech MCEP features and the standard sound dataset, so that the trained CycleGAN-VC2 network model is more inclined to the conversion of the target speech. Then the standard sound of the text to be converted is input into the trained CycleGAN-VC2 network model and processed to obtain the required speech, thereby greatly improving the similarity between the converted speech and the target speech, and realizing speech generation of any speaker. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 A schematic diagram of a workflow of a speech generation method for non-parallel speech conversion provided by an embodiment of the present invention;

[0042] Figure 2 A schematic diagram of the position of adding a self-attention mechanism to a residual network of a model generator of a speech generation method for non-parallel speech conversion provided by an embodiment of the present invention;

[0043] Figure 3 A schematic diagram of the position of adding a self-attention mechanism to a model discriminator network of a speech generation method for non-parallel speech conversion provided by an embodiment of the present invention;

[0044] Figure 4 The present invention provides a method for generating speech for non-parallel speech conversion provided by an embodiment of the present invention, and a schematic diagram of a speech generator produced by using PyQt5. DETAILED DESCRIPTION

[0045] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention, so the present invention is not limited by the specific embodiments disclosed below.

[0046] See also Figure 1 to Figure 3 The embodiment of the present invention provides a speech generation method for non-parallel speech conversion, comprising the following steps:

[0047] Step 1: Use news, essays and other text as materials, use pyttsx3 to generate male and female standard voice datasets, and obtain the speech dataset of the target speaker.

[0048] Step 2: Extract the logarithmic fundamental frequency mean, variance, non-periodic features, and MCEP feature information of the standard voice and the target voice.

[0049] The MCEP feature is a variant based on the Mel-frequency cepstral coefficients (MFCC). Compared with MFCC, MCEP focuses more on describing the short-time spectral characteristics of speech. The MCEP feature extracts the Mel-frequency cepstral coefficients MFCC of the speech signal, converts it to the Mel-frequency scale, and further calculates its cepstral to obtain coefficients that can reflect the characteristics of the speech signal. This coefficient can capture the prosodic features, speaking characteristics, language information, language phonemes, speech pitch and speech intonation of the speech.

[0050] Step 3: Send the extracted MCEP features of the standard sound and the target speech into the improved CycleGAN-VC2 model for training to obtain the corresponding model for converting the standard sound features to the target speech.

[0051] Step 4: Input text, pyttsx3 generates the standard speech of the input text content, and uses PyWORLD to extract the fundamental frequency f0, spectrum envelope, non-periodic features, and MCEP feature information of the generated standard speech.

[0052] Step 5: Input the standard speech MCEP features extracted in step 2 into the conversion model trained in step 3, and output the converted MCEP features.

[0053] Step 6: Use the logarithmic fundamental frequency mean and variance information of the standard sound and target speech extracted in step 2 to transform the standard fundamental frequency f0 extracted in step 4.

[0054] Step 7: Use PyWORLD to synthesize the target speaker's speech by combining the converted fundamental frequency f0' obtained in step 6, the non-periodic features of the target speech extracted in step 2, and the MCEP features converted in step 5.

[0055] In step six, the baseband function is converted to:

[0056]

[0057] Where: f′0 is the converted fundamental frequency; μ′ and σ′ are the mean and mean square error of the converted fundamental frequency in the logarithmic domain; μ and σ are the mean and mean square error of the source speech fundamental frequency in the logarithmic domain; f0 is the source speech fundamental frequency.

[0058] In step 3, the improved CycleGAN-VC2 structure and its training loss function are:

[0059] The improved CycleGAN-VC2 model consists of a generator and a discriminator. The generator consists of a downsampling layer (2D to 1D), a residual layer, a self-attention mechanism, and an upsampling layer (1D to 2D). The self-attention mechanism is deployed in the middle of the 6-layer residual module. The discriminator adopts the PatchGAN structure. PatchGAN is a 2D convolutional neural network used to discriminate 2D spectral texture data. It is mainly composed of 1D CNN, 2D CNN, GLU layer and Instance normalization layer. The downsampling layer consists of 1D CNN, Instance normalization layer and GLU linear gating unit. Multi-layer downsampling can further reduce the size of the input signal, making it convenient to process larger data sets in the lower layer network. The discriminator self-attention mechanism is deployed after the first layer of downsampling. The specific implementation method is to extract Q, K, and V information through three different convolutional layers. The calculation formula of the self-attention mechanism is:

[0060]

[0061] Among them: Q (Query), K (Key), V (Value) are the input query, key and value respectively, representing the original input information.

[0062] In the translation task, Query can be regarded as a sequence of original word vectors, while Key and Value can be regarded as a sequence of target word vectors. k It is the input dimension of Query and Key; first calculate the dot product of Query and all Keys, i.e. QK T , represents the similarity score between the query and the key, which is used to measure the relevance of the query to each key; each is then divided by To prevent the product result from being too large, it is then fed into the Softmax function.

[0063] The improved CycleGAN-VC2 includes 2 generators: G X→Y and G Y→X , 4 discriminators: D X , D Y and D′ for generating the quadratic adversarial loss X and D′ Y ; The total loss function consists of primary adversarial loss, secondary adversarial loss, cycle consistency loss and identity mapping loss, and its composition formula is as follows:

[0064] L=L adv +L adv2 +λ cyc L cyc (G X→Y , G Y→X )+λ id L id (G X→Y, G Y→X )

[0065] Where: cyc , id are the hyperparameters of the cycle consistency loss and identity mapping loss, respectively, and adjust the weights of the corresponding losses during training.

[0066] The first adversarial loss is used to measure the similarity between the converted speech features of the generator and the real speech features. The smaller the loss value, the more similar the converted speech features are to the real speech features. The adversarial losses of the discriminators D_x and D_y are:

[0067]

[0068]

[0069] Where: PX(x) and PY(y) are the distributions of source speech data and target speech data respectively.

[0070] The adversarial loss is:

[0071] L adv =L adv (G X→Y , D Y )+L adv (G Y→X , D X )

[0072] Where: L adv Indicates a loss of confrontation.

[0073] The purpose of adding quadratic adversarial loss is to alleviate the over-smoothing effect caused by cycle consistency loss.

[0074] Since the adversarial loss is used twice in each cycle during training, it is called quadratic adversarial loss:

[0075]

[0076] L adv2 =L adv2 (G X→Y , G Y→X , D′ X )+L adv2 (G Y→X , G X→Y , D′ Y )

[0077] Where: L adv2 Denotes the quadratic adversarial loss.

[0078] The purpose of the cycle consistency loss is to ensure that the speech content is preserved when converting speech features. This allows the audio generated by the model to remain similar to the original audio after forward and reverse conversion (i.e., from X to Y, and then from Y to X). The formula is:

[0079]

[0080] Where: L cyc represents the cycle consistency loss.

[0081] When the generator does not need to convert the speech, in order to make the speech pass through the generator without change and keep the identity information consistent with the original speech as much as possible, the identity mapping loss should be added during the generator training. The formula is:

[0082]

[0083] Where: L id represents the identity mapping loss.

[0084] The present invention utilizes the text-to-speech generation function of pyttsx3 combined with the speech conversion function of the CycleGAN-VC2 model to realize the speech generation from text to any speaker, providing a new method for the design of a speech generator.

[0085] The present invention adds self-attention mechanisms to the generator and discriminator parts respectively, thereby enhancing the feature learning ability of the generator residual module and achieving the effect of improving speech quality and clarity; adding a self-attention mechanism to the discriminator enables the discriminator to more accurately distinguish the authenticity of weighted features.

[0086] The speech generation method of the present invention does not require parallel speech data sets, thus reducing the production cost of the data sets and being more convenient for application and promotion.

[0087] like Figure 4 As shown in the figure, the speech generator effect of the present invention using PyQt5 is schematically shown, which specifically includes:

[0088] Step 1: pyttsx3 male standard voice id uses: *BEEY_IOCAL_MACHINE\SOFTWARE\Microsoft\speech\Voices\Tokens\MSTT s_vl10_zhCN_KangkangH'; female standard voice id uses: 'HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Speech\Voices\Tokens\TTS_MS_ZH-CN_HUIHUI_11.0'.

[0089] The male standard voice dataset and the female standard voice dataset were generated from mobile phone text content such as news and essays. The male standard voice training dataset consists of 110 items with a total duration of 12.5 minutes and an average duration of 6.86 seconds per voice item. The female standard voice training dataset consists of 200 items with a total duration of 15.6 minutes and an average duration of 8.49 seconds per voice item. The target male and female voice datasets are the voice datasets selected from the CycleGAN-VC2 project in YuHong-LDU / SpeechProcessing on the gitee platform, with a total of 110 items, a total duration of 9 minutes and an average duration of 4.89 seconds. The BZNSYP female voice dataset contains 10,000 Chinese female voices with a total duration of about 12 hours. 200 voices are selected as the target female voice training dataset, with a total duration of 13.6 minutes and an average duration of 4.09 seconds per item.

[0090] Step 2: In the preprocessing stage, the speech generator uses the WORLD vocoder to extract the MCEP features, logarithmic fundamental frequency features and non-periodic features of the source speaker and the target speaker before training the speech conversion model and saves the file. The sampling rate is set to 16000, the frame shift is 5 milliseconds, and 36-dimensional MCEP features are extracted; when preprocessing the experimental data, the variance and mean of the training data are used to normalize the MCEP features.

[0091] Step 3: During the training process, in order to ensure the randomness of the training data, 128 consecutive frames are randomly extracted for each speech; the Adam optimizer is used for gradient update, the exponential decay rate β1 of the first-order moment estimation is 0.5, the exponential decay rate β2 of the second-order moment estimation is 0.999, the generator learning rate is 0.0002, the discriminator learning rate is 0.0001, the number of iterations is 20000, the batch size is 1, and the total loss of the generator is λ cyc and λ id Set to 5 and 10 respectively.

[0092] Step 4: In voice conversion, the same-sex voice conversion effect is better than the opposite-sex conversion, so the target speaker's voice should be the same as the standard voice in terms of gender; after the model training is completed, Figure 4 As shown, enter text in the "Text Input", select male / female standard voice, select the trained speech model and the speech spectrum envelope and non-periodic feature file extracted by the target speaker's speech preprocessing, enter the speaking speed and volume and click Generate. The text first passes through pyttsx3 to generate male / female standard voice with a certain speaking speed and volume, and then the MCEP features are extracted by the WORLD vocoder and sent to the conversion model to convert the target speech MCEP features, and then the WORLD vocoder is used to synthesize the speech, and the play sound library is used to play it.

[0093] The present invention uses an improved CycleGAN-VC2 network in the speech conversion part, adds a self-attention mechanism in the middle of the residual layer of the generator and after the first layer of the discriminator is sampled to improve the speech conversion similarity, and combines the pyttsx3 text-to-speech generation function to realize speech generation of any speaker.

[0094] The above-mentioned embodiment only expresses one implementation mode of the present invention, and its description is relatively specific and detailed, but it cannot be understood as limiting the scope of the invention patent. It should be pointed out that for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be based on the attached claims.

Claims

1. A speech generation method for non-parallel speech conversion, characterized in that: The following steps are involved: Obtain the target's speech recording data and form the target's speech data set; at the same time obtain the standard sound data set; Constructing a CycleGAN-VC2 network model, respectively adding a first self-attention mechanism between the multi-layer residual modules of the CycleGAN-VC2 network model generator to form an improved generator; adding a second self-attention mechanism after the downsampling first layer of the CycleGAN-VC2 network model discriminator to form an improved discriminator; and forming an improved CycleGAN-VC2 network model; Extracting the target speech MCEP features in the target speech data set and the standard speech MCEP features in the standard speech data set; The improved CycleGAN-VC2 network model is trained using the target speech MCEP features and the standard speech MCEP features. During the training, the generator is first fixed and the discriminator is trained. At the same time, the discriminator is fixed and the generator is trained to obtain the trained CycleGAN-VC2 network model. Convert the text to be generated into standard pronunciation, extract the MCEP features of the standard pronunciation, and input the MCEP features of the standard pronunciation into the generator of the trained CycleGAN-VC2 network model to obtain the converted text MCEP features; Extract the fundamental frequency of the standard text sound and convert it to extract the non-periodic features of the target speech data set; The desired speech is synthesized by combining the converted text MCEP features, the converted text standard tone fundamental frequency and the non-periodic features of the target speech data set.

2. A method for generating speech for non-parallel speech conversion according to claim 1, characterized in that: The acquisition of the standard sound data set includes: Using news and prose text as materials, and using pyttsx3 to generate male and female standard voice datasets.

3. A speech generation method for non-parallel speech conversion according to claim 1, characterized in that: The improved generator comprises: The improved generator is a cascade of downsampling layers, residual layers, self-attention mechanisms, and upsampling layers; The downsampling layer is 2D to 1D, and the upsampling layer is 1D to 2D; The self-attention mechanism is deployed between the 6 residual layers.

4. A method for generating speech for non-parallel speech conversion according to claim 1, characterized in that: The improved discriminator comprises: The improved discriminator is a cascade of 1D CNN, 2D CNN, GLU layer and Instance normalization layer; The 1D CNN, Instance normalization layer and GLU layer in the discriminator constitute the downsampling layer of the discriminator; A self-attention mechanism is deployed after the first layer of the downsampling layer.

5. A method for generating speech for non-parallel speech conversion according to claim 1, characterized in that: The process of extracting features through the self-attention mechanism includes: The self-attention mechanism extracts the input query Query, input key Key, and input value Value information through three different convolutional layers; Since the weight parameters of the three convolutional layers are different, when extracting information from the same data, the extracted feature vectors will be different. The extracted feature vectors are obtained using the self-attention mechanism to generate a weighted summation result. The way the self-attention mechanism extracts features is as follows: Where: Q is the input query; K is the input key; V is the input value; d k The input dimensions of Query and Key; QK T Score the similarity between the query and the key.

6. A method for generating speech for non-parallel speech conversion according to claim 1, characterized in that: The improved CycleGAN-VC2 network model includes: The improved CycleGAN-VC2 network model includes 2 generators and 4 discriminators; The 2 generators are: G X→Y and G Y→X ; The 4 discriminators are: D X , D Y and D′ for generating the quadratic adversarial loss X and D′ Y ; The total loss function of the improved CycleGAN-VC2 network model consists of primary adversarial loss, secondary adversarial loss, cycle consistency loss and identity mapping loss, and its formula is: L=L adv +L adv2 +λ cyc L cyc (G X→Y ,G Y→X )+λ id L id (G X→Y ,G Y→X ) Where: L is the total loss function; L adv For a loss against; L adv2 is the secondary confrontation loss; L cyc (G X→Y , G Y→X ) is the cycle consistency loss; λ cyc is the hyperparameter of cycle consistency loss; L id (G X→Y , G Y→X ) is the identity mapping loss; λ id Hyperparameters for the identity mapping loss.

7. A method for generating speech for non-parallel speech conversion according to claim 1, characterized in that: The step of extracting and converting the fundamental frequency of the standard pronunciation of the text includes: Use PyWORLD to extract the logarithmic fundamental frequency mean, variance, non-periodic features and MCEP features of the target speech dataset and the standard sound dataset; PyWORLD is used to extract the fundamental frequency f0 of the standard text sound. According to the logarithmic fundamental frequency mean and variance of the target speech dataset and the standard sound dataset, the fundamental frequency f0 of the standard text sound is converted to obtain the converted fundamental frequency f0'. The conversion method is: Where: f0′ is the converted fundamental frequency; μ′ and σ′ are the mean and mean square error of the converted fundamental frequency in the logarithmic domain; μ and σ are the mean and mean square error of the fundamental frequency of the source speech in the logarithmic domain; f0 is the fundamental frequency of the source speech.

Images