A many-to-one speech conversion method based on gammatone frequency cepstral coefficients

Abstract

The present invention relates to a many-to-one speech conversion method based on gammatone frequency cepstral coefficients. The method comprises the following steps: extracting gammatone frequency cepstral coefficient features of a speech sample to be converted, inputting the features into a trained speech content extraction model to obtain a semantic posterior probability map of the speech sample to be converted, inputting the features into a trained CBHG model to obtain gammatone cepstral coefficients of the speech sample to be converted; and inputting the gammatone cepstral coefficients of the speech sample to be converted into a trained neural vocoder to obtain the converted speech. Compared with existing technologies, the present invention has the advantage of higher speech conversion clarity.

Description

Technical Field

[0001] The present invention relates to the field of speech conversion, and in particular to a many-to-one speech conversion method based on gammatone frequency cepstral coefficients. Background Art

[0002] Voice conversion technology can transform the timbre, emotion, and accent of a source speaker into that of a target speaker without changing the content of the speech. Potential applications include voice camouflage, personalized voice synthesis, and voiceprint unlocking.

[0003] Speech conversion technology requires that the converted speech has good naturalness and intelligibility. At the same time, it should be highly similar to the target speaker's voice in terms of timbre, emotion, accent, etc.

[0004] To ensure the quality of speech conversion, traditional speech conversion models are mostly trained using parallel data. Parallel data refers to paired data samples consisting of the speech signals of the source and target speakers when they utter the same content. However, collecting high-quality parallel data is difficult, time-consuming, and expensive, significantly limiting its application. Furthermore, speech conversion requires time alignment. Due to technical limitations, this alignment process inevitably introduces errors, which impacts the quality of speech conversion. Currently, most speech conversion models use Mel-Frequency Cepstral Coefficients (MFCCs) as input features. However, the frequency selectivity of Mel filter banks does not accurately simulate that of the basilar membrane of the human ear, thus affecting the quality of synthesized speech.

[0005] Furthermore, when synthesizing speech using the target speaker's pitch and related parameters, parametric vocoders, such as the STRAIGHT vocoder, are often used. The parameters of these parametric vocoders must be manually tuned based on oversimplified assumptions in signal processing. Consequently, they can only provide suboptimal solutions. Furthermore, the emergence of generative adversarial networks, such as StarGAN, has significantly advanced speech conversion. However, while these models achieve relatively good results and can perform many-to-many speech conversion, they can only convert the speech of speakers that appear in the training set. Summary of the Invention

[0006] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a many-to-one speech conversion method based on gammatone frequency cepstral coefficients.

[0007] The purpose of the present invention can be achieved by the following technical solutions:

[0008] A many-to-one speech conversion method based on gammatone frequency cepstral coefficients comprises the following steps:

[0009] S1. Extract the gammatone frequency cepstral coefficient features (GFCC) of the speech sample to be converted, input the gammatone frequency cepstral coefficient features into the trained speech content extraction model to obtain the semantic posterior probability map (Phonetic Posterior Grams, PPGs) of the speech sample to be converted, input the semantic posterior probability map into the trained CBHG model to obtain the gammatone cepstral coefficients (GCEP) of the speech sample to be converted;

[0010] S2, inputting the gamma-tone cepstral coefficients of the speech sample set to be converted into the trained neural vocoder to obtain the converted speech;

[0011] The training process of the speech content extraction model is as follows:

[0012] A1. Obtain a first training sample set from a Chinese TTS corpus and annotate the first training sample set;

[0013] A2. Extracting gammatone frequency cepstral coefficient features of samples in the first training sample set and inputting them into a speech content extraction model for training;

[0014] The training process of the CBHG model is as follows:

[0015] B1. After obtaining the trained speech content extraction model, obtain a second training set of samples from the target speaker corpus, extract the gammatone frequency cepstral coefficient features of the second training set samples, and input them into the speech content extraction model to obtain a semantic posterior probability map of the second training sample set.

[0016] B2. Extract the gamma-tone cepstral coefficients of the samples in the second training sample set, use the semantic posterior probability map of the second training sample set as the input of the CBHG model, use the gamma-tone cepstral coefficients of the samples in the second training sample set as the target output of the CBHG model, and train the CBHG model;

[0017] The training process of the neural vocoder is as follows:

[0018] The gammatone cepstral coefficients of the samples in the second training sample set are used as the input of the neural vocoder, and the original speech of the second training sample set is used as the target output of the neural vocoder to train the neural vocoder.

[0019] Furthermore, the method for labeling the first training sample set in step A1 includes: automatically labeling phoneme boundaries using a speech-aligner tool based on machine learning.

[0020] Furthermore, the method for extracting the gammatone frequency cepstral coefficient feature includes:

[0021] C1. Pre-emphasize and frame the voice signal;

[0022] C2. Perform short-time Fourier transform on the framed signal;

[0023] C3. Inputting the amplitude spectrum of the signal after short-time Fourier transform into a gamma-tone filter bank to obtain a filtered signal;

[0024] C4, extract the cube root of the filtered signal as the gamma-tone cepstral coefficient;

[0025] C5. Perform discrete cosine transform on the gammatone frequency cepstral coefficient to obtain gammatone frequency cepstral coefficient features.

[0026] Furthermore, the gammatone frequency cepstral coefficient features are obtained and then normalized.

[0027] Furthermore, the normalized gamma-tone frequency cepstral coefficient feature f GFCC_norm The formula is as follows:

[0028]

[0029] Among them, f min Represents the matrix composed of the minimum values in the gamma-tone frequency cepstral coefficient characteristics, f max A matrix representing the maximum values of the gammatone frequency cepstral coefficient features.

[0030] Furthermore, the neural vocoder includes a generator and a discriminator, wherein the generator is used to generate a speech signal, and the discriminator is used to determine the authenticity of the generated signal.

[0031] Furthermore, the loss function L of the generator is adv The expression of (G, D) is as follows:

[0032] L adv (G, D) = E z～N(0，I) [(1-D(G(z))) 2 ]

[0033] Among them, z represents the input white noise, G represents the generator, D represents the discriminator, and E represents the expectation.

[0034] Furthermore, the objective function L of the discriminator DThe expression of (G, D) is as follows:

[0035]

[0036] Among them, z represents the input white noise, G represents the generator, D represents the discriminator, E represents the expectation, x and P x Represent the target speech waveform and its corresponding distribution respectively.

[0037] Furthermore, the multi-resolution short-time Fourier transform loss function L used by the neural vocoder is aux (G), the expression is as follows:

[0038]

[0039] Among them, λ represents the number of short-time Fourier transform losses, and the expression of a single short-time Fourier transform loss is as follows:

[0040]

[0041]

[0042]

[0043] in,‖.‖ F represents Frobenius regularization, ‖.‖1 represents L1 regularization, z represents the input white noise, Pz represents the distribution of white noise waveform, G represents the generator, D represents the discriminator, E represents the expectation, x and P x Represent the target speech waveform and its corresponding distribution respectively, STFT represents short-time Fourier transform, N represents the number of frames, Represents the generated speech waveform samples.

[0044] Furthermore, the first training sample set and the second training sample set need to be preprocessed, and the preprocessing includes denoising and adjusting the sampling rate to 16 kHz.

[0045] Compared with the prior art, the present invention has the following advantages:

[0046] 1. The present invention mainly relates to a speech content extraction model, a CBHG model, and a neural vocoder, all of which are trained around the gammatone frequency cepstral coefficient feature. First, because this feature well simulates the frequency selection characteristics of the basilar membrane of the human ear, the clarity and naturalness of the converted speech are improved; second, due to the introduction of the neural vocoder, the model's adaptability to data is improved, and the inference speed is also improved; finally, because the Chinese TTS corpus contains information on multiple speakers, multiple speakers can be used to convert the target speaker's speech through a single training session.

[0047] 2. The present invention uses a machine learning-based speech-aligner to mark phoneme boundaries in speech, thereby greatly reducing the requirements for the training corpus and improving the adaptability of the model.

[0048] 3. The present invention normalizes the gammatone frequency cepstral coefficient characteristics to facilitate data processing and accelerate the convergence speed of model training. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 Schematic diagram of the speech conversion process of the present invention.

[0050] Figure 2 This is a complete flow chart of the present invention including the training model. DETAILED DESCRIPTION

[0051] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0052] This embodiment provides a many-to-one speech conversion method based on gammatone frequency cepstral coefficients, such as Figure 1 As shown, the specific steps include:

[0053] Step S1, extracting gamma-tone frequency cepstral coefficient features of the speech sample to be converted, inputting the gamma-tone frequency cepstral coefficient features into a trained speech content extraction model to obtain a semantic posterior probability map of the speech sample to be converted, and inputting the semantic posterior probability map into a trained CBHG model to obtain gamma-tone cepstral coefficients of the speech sample to be converted;

[0054] Step S2: inputting the gamma-tone cepstral coefficients of the speech sample to be converted into a trained neural vocoder to obtain the converted speech;

[0055] The training process of the speech content extraction model can be referred to Figure 2 The contents of the first column are as follows:

[0056] Step A1: Obtain a first training sample set from a Chinese TTS corpus containing corpora from multiple speakers and annotate the first training sample set. In this embodiment, a machine learning-based speech-aligner is used as an annotation tool to complete phoneme annotation. The annotated audio and labels are segmented into 2-second segments to improve data utilization. The phoneme annotation formula is as follows:

[0057] F(x i )=P j (t s , t e)

[0058] Where F(·) represents the labeling function. j represents the jth element in the phoneme table, t s , t e Respectively represent the start time and end time of the phoneme.

[0059] Step A2: extract the gamma-tone frequency cepstral coefficient features of the samples in the first training sample set and input them into the speech content extraction model for training. The learning rate is 0.0003 and the number of training iterations is 1000. The steps for extracting the gamma-tone frequency cepstral coefficient features are as follows:

[0060] Step C1: pre-emphasize and frame the speech signal, wherein the number of frames N is calculated according to the following formula:

[0061]

[0062] Among them, T length Indicates the total length of the speech signal, T win Indicates the frame length. In this embodiment, it is preferably 0.0375s. hop represents the frame shift, which is preferably 0.016s in this embodiment.

[0063] The expression of the window function w[n] used for framing is as follows:

[0064]

[0065] Here, M represents the window length of the Hanning window, which is preferably 600 sampling points in this embodiment.

[0066] Step C2: Perform short-time Fourier transform on the framed signal. The Fourier transform result X(k) of the framed digital signal sequence is:

[0067]

[0068] Wherein, x(n) represents a digital signal sequence, and L represents the Fourier transform accuracy, which is preferably 1024 in this embodiment.

[0069] Step C3: Input the amplitude spectrum of the signal |X(k)| after the short-time Fourier transform into the gamma-tone filter bank to obtain the filtered signal γ i (k)*|X(k)|. Among them, γ i (k) represents the spectrum of the gammatone filter bank, and the corresponding impulse response g i (t) is expressed as follows:

[0070]

[0071] Among them, A represents the gain of the filter, P represents the filter order, and f i represents the center frequency, represents phase, and ERB represents equivalent rectangular bandwidth. The calculation expression of equivalent rectangular bandwidth is as follows:

[0072]

[0073] Step C4: Take the cube root of the filtered signal as the gamma-tone frequency cepstral coefficient f GCEP (k).

[0074]

[0075] Step C5: Perform discrete cosine transform on the gamma-tone frequency cepstral coefficients to obtain the gamma-tone frequency cepstral coefficient feature f GFCC (n), the calculation expression is as follows:

[0076]

[0077]

[0078] Where τ represents the generalized frequency, μ represents the scale factor, and K represents the number of signal points.

[0079] For ease of calculation, the gammatone frequency cepstral coefficients are normalized using the following formula in this embodiment. The calculation expression is as follows:

[0080]

[0081] Among them, f min Represents the matrix composed of the minimum values in the gamma-tone frequency cepstral coefficient characteristics, f max A matrix representing the maximum values of the gammatone frequency cepstral coefficient features.

[0082] In this embodiment, the training process of the CBHG model can refer to Figure 2 The specific steps are as follows:

[0083] Step B1: After obtaining the trained speech content extraction model, obtain a second training sample set from the target speaker corpus, extract the gammatone frequency cepstral coefficient features of the samples in the second training sample set, and input them into the speech content extraction model to obtain the semantic posterior probability map of the samples in the second training sample set.

[0084] Step B2: extract the gamma-tone cepstral coefficients of the samples in the second training sample set, use the semantic posterior probability map of the samples in the second training sample set as the input of the CBHG model, use the gamma-tone cepstral coefficients of the second training samples as the target output of the CBHG model, and train the CBHG model with a learning rate of 0.0003, a frame shift of 0.016s, a frame length of 0.0375s, a Fourier transform accuracy of 1024, and 5000 iterations.

[0085] In this embodiment, the neural vocoder is a PARALLEL WaveGAN neural vocoder. This is a non-autoregressive vocoder that can achieve better speech conversion effects. Its training process can be referred to Figure 2 The specific steps are as follows:

[0086] The gammatone cepstral coefficients of the samples in the second training sample set are used as the input of the neural vocoder, and the original speech of the samples in the second training sample set is used as the target output of the neural vocoder to train the neural vocoder.

[0087] The vocoder network consists of two parts: the generator (G) and the discriminator (D). The generator is used to learn the real waveform distribution and generate signals, and the discriminator is used to determine whether the generated signal is a real sample of the target person. The loss function of the generator is L adv The expression of (G, D) is as follows:

[0088] L adv (G,D)=E z～N(0,I) [(1-D(G(z))) 2 ]

[0089] Among them, z represents the input white noise, G represents the generator, D represents the discriminator, and E represents the expectation.

[0090] The objective function expression of the discriminator is L D (G,D) are as follows:

[0091]

[0092] Among them, x and P x Represent the target speech waveform and its corresponding distribution respectively.

[0093] In order to increase the stability of the PARALLEL WaveGAN neural vocoder and the efficiency of adversarial training, a multi-resolution short-time Fourier transform loss function L is introduced into the neural vocoder. aux (G), the expression is as follows:

[0094]

[0095] Among them, λ represents the number of short-time Fourier transform losses, and the expression of a single short-time Fourier transform loss is as follows:

[0096]

[0097]

[0098]

[0099] in,‖.‖ F represents Frobenius regularization, ‖.‖1 represents L1 regularization, STFT represents short-time Fourier transform, Represents the generated speech waveform samples.

[0100] In this embodiment, both the first training set samples and the second training set samples used need to be preprocessed, including denoising and adjusting the sampling rate to 16 kHz.

[0101] The above describes in detail the preferred embodiments of the present invention. It should be understood that those skilled in the art can make numerous modifications and variations based on the concepts of the present invention without inventive effort. Therefore, any technical solutions that can be derived by those skilled in the art through logical analysis, reasoning, or limited experimentation based on the concepts of the present invention and the prior art should be within the scope of protection defined by the claims.

Claims

1. A many-to-one speech conversion method based on gammatone frequency cepstral coefficients, characterized in that: The following steps are involved: S1. Extract gamma-tone frequency cepstral coefficient features of the speech sample to be converted, input the gamma-tone frequency cepstral coefficient features into a trained speech content extraction model to obtain a semantic posterior probability map of the speech sample to be converted, input the semantic posterior probability map into a trained CBHG model to obtain gamma-tone cepstral coefficients of the speech sample to be converted; S2, inputting the gamma-tone cepstral coefficients of the speech sample to be converted into the trained neural vocoder to obtain the converted speech; The training process of the speech content extraction model is as follows: A1. Obtain a first training sample set from a Chinese TTS corpus and annotate the first training sample set; A2. Extracting gammatone frequency cepstral coefficient features of samples in the first training sample set and inputting them into a speech content extraction model for training; The training process of the CBHG model is as follows: B1. After obtaining the trained speech content extraction model, obtain a second training sample set from the target speaker corpus, extract the gammatone frequency cepstral coefficient features of the samples in the second training sample set, and input them into the speech content extraction model to obtain a semantic posterior probability map of the samples in the second training sample set; B2. Extract the gamma-tone cepstral coefficients of the samples in the second training sample set, use the semantic posterior probability map of the second training sample set as the input of the CBHG model, use the gamma-tone cepstral coefficients of the samples in the second training sample set as the target output of the CBHG model, and train the CBHG model; The training process of the neural vocoder is as follows: Using the gammatone cepstral coefficients of the samples in the second training sample set as the input of the neural vocoder, using the original speech of the samples in the second training sample set as the target output of the neural vocoder, and training the neural vocoder; The method for extracting the gamma-tone frequency cepstral coefficient feature includes: C1. Pre-emphasize and frame the voice signal; C2. Perform short-time Fourier transform on the framed signal; C3. Inputting the amplitude spectrum of the signal after short-time Fourier transform into a gamma-tone filter bank to obtain a filtered signal; C4, extract the cube root of the filtered signal as the gamma-tone cepstral coefficient; C5. Perform discrete cosine transform on the gammatone cepstral coefficients to obtain gammatone frequency cepstral coefficient features.

2. The many-to-one speech conversion method based on gammatone frequency cepstral coefficients according to claim 1, characterized in that: The method for labeling the first training sample set in step A1 includes: automatically labeling phoneme boundaries using a speech-aligner tool based on machine learning.

3. The many-to-one speech conversion method based on gammatone frequency cepstral coefficients according to claim 1, characterized in that: Get the gammatone frequency cepstral coefficient features and normalize them.

4. The many-to-one speech conversion method based on gammatone frequency cepstral coefficients according to claim 3, characterized in that: The normalized gamma-tone frequency cepstral coefficient feature f GFCC_norm The formula is as follows: Among them, f min Represents the matrix composed of the minimum values in the gamma-tone frequency cepstral coefficient characteristics, f max Represents the matrix composed of the maximum values in the gamma-tone frequency cepstral coefficient characteristics, f GFCC Represents the gammatone frequency cepstral coefficient characteristics.

5. The many-to-one speech conversion method based on gammatone frequency cepstral coefficients according to claim 1, characterized in that: The neural vocoder includes a generator and a discriminator, wherein the generator is used to generate a speech signal, and the discriminator is used to determine the authenticity of the generated speech signal.

6. The many-to-one speech conversion method based on gammatone frequency cepstral coefficients according to claim 1, characterized in that: The first training sample set and the second training sample set need to be preprocessed, and the preprocessing includes denoising and adjusting the sampling rate to 16 kHz.

Images