Enhanced decoupling-based iteratively optimized multi-component speech conversion method

Abstract

The invention discloses an iteratively optimized multi-component voice conversion method based on enhanced decoupling, and the method comprises the steps: respectively extracting rhythm, content and tone information in voice and pitch information in a pitch contour through employing a multi-channel encoder group with an information bottleneck, and generating the voice through employing a decoder in combination with the encoder group; random resampling and pitch enhancement are adopted to selectively eliminate rhythm or pitch information flowing into an encoder, and then the speech de-entanglement capability of the model is enhanced through a multi-task training method of cooperative pitch contour reconstruction, adversarial speaker classifier group, content feature reconstruction and content feature-rhythm feature mutual information minimization. Besides, the training process is simplified through a two-stage method, the convergence efficiency of the model is improved, and the method aims at achieving high-quality multi-component voice conversion so as to solve the problem that a traditional voice conversion method can only decouple timbre and other voice components are still entangled together.

Description

Technical Field

[0001] The invention belongs to the technical field of speech processing, and in particular relates to an iteratively optimized multi-component speech conversion method based on enhanced decoupling. Background Art

[0002] Human language is a treasure trove of information that consists of four fundamental components: content, which conveys semantics, pitch and rhythm, which shape the emotional tone of speech through the frequency and length of syllables, and timbre, which is a marker of speaker identity. Many speech applications are designed to target these specific components, for example: automatic speech recognition programs are designed to transcribe speech content into text, automatic speaker verification can extract timbre, and singing voice synthesis programs manipulate pitch and rhythm. However, the challenge of disentangling these entangled components has sparked widespread interest and effort.

[0003] Traditional speech conversion technology attempts to separate the speaker's information from the source speech while retaining high-level factors such as content and emotion to keep the speech consistent with the timbre of the target speaker. Previous studies have achieved better results by incorporating prior settings into the model structure. For example, the literature [Lifa Sun, Kun Li, Hao Wang, Shiyin Kang, and Helen M. Meng, "Phonetic posteriorgrams for many-to-one voice conversion without parallel data training," in IEEE International Conference on Multimedia and Expo, ICME 2016, Seattle, WA, USA, July 11-15, 2016.2016, pp. 1–6, IEEE Computer Society] proposed using the phonetic posteriorgrams (PPGs) of the pre-trained speech recognition model as speaker-independent information to decouple timbre; another example is the literature [Xintao Zhao, Feng Liu, Changhe Song, Zhiyong Wu, Shiyin Kang, Deyi Tuo, and Helen Meng, "Disentangling content and fine-grained prosody information via hybrid ASR bottleneck features for voice conversion," in Proc.ICASSP.2022, pp.7022–7026, IEEE] proposed to use a bottleneck encoder (Bottleneck Encoder, BNE) to force decoupling of timbre by controlling the representation freedom of the feature latent space. However, although these traditional speech conversion technologies have achieved certain results, they are still troubled by entangled factors such as pitch and rhythm, resulting in the generated converted audio may have rhythmic mutations and other phenomena, and the conversion effect is not good; in addition, the existing technology can only decouple and edit the timbre information in the speech, and it is difficult to effectively control the pitch and rhythm information in the speech.

[0004] In order to solve the above problems, the literature [Kaizhi Qian, Yang Zhang, Shiyu Chang, Mark Hasegawa-Johnson, and David Cox, "Unsupervised speech decomposition via triple information bottleneck," in Proceedings of the 37th International Conference on Machine Learning, 2020, pp. 7836–7846] proposed SpeechSplit, which was the first to try the multi-component decoupling transformation involving the timbre, pitch, rhythm, and content of speech. It uses a multi-channel bottleneck encoder group to extract the content, timbre, and rhythm information in the Mel spectrum and the pitch information in the pitch contour, and uses random resampling technology to destroy the rhythm information in the Mel spectrum to assist in improving the encoder's ability to extract the corresponding information. However, the decoupling effect of SpeechSplit is heavily dependent on the design of the bottleneck parameters in the bottleneck encoder. The parameters need to be fine-tuned for data in different scenarios and training sets, which will consume a lot of manpower and computing resources in actual use.

[0005] Therefore, we urgently need a robust, high-quality speech conversion method that can fully decouple the various components of speech, in order to address the deficiencies of traditional methods in terms of pitch, rhythm, etc., while improving the robustness of the decoupling capability under different data distributions, thereby achieving better multi-component speech conversion effects. Specifically, the present invention proposes a novel multi-component speech conversion system that can perform high-quality disentanglement and conversion of the rhythm, pitch, and timbre in speech with a single sample; at the same time, a two-stage training strategy is designed to accelerate the convergence of the model while improving the model effect; in addition, the present invention uses pwCCA to evaluate the decoupling capabilities of the various components of the model, and according to the evaluation results, resets the weighted values ​​of the loss function, and retrains the model for iterative optimization. Summary of the invention

[0006] In view of the above, the present invention provides an iteratively optimized multi-component speech conversion method based on enhanced decoupling. By using a priori random resampling, pitch enhancement and two-stage decoupling training techniques, it solves the problem that the traditional timbre conversion model can only decouple the timbre while other speech components are still entangled, resulting in the generated speech rhythm jumps and uncontrollable problems. The model can perform high-quality arbitrary conversion of the timbre, pitch and rhythm components in the speech.

[0007] A multi-component speech conversion method based on enhanced decoupling and iterative optimization comprises the following steps:

[0008] (1) Obtain a speech dataset, perform pitch enhancement processing on each speech, and extract the pitch contour and Mel spectrogram;

[0009] (2) Construct a multi-component speech conversion model, which includes:

[0010] Joint random resampling module, used to perform joint random resampling of pitch contour and pitch-enhanced Mel spectrogram in the time dimension;

[0011] A multi-channel encoder group with information bottleneck extracts speech component features corresponding to content, rhythm, timbre, and pitch based on pitch contours and Mel spectrograms;

[0012] Mel spectrogram decoder, combining all speech component features to reconstruct a new Mel spectrogram X rec ;

[0013] Auxiliary pitch contour decoder, which reconstructs the speech component features of pitch and rhythm to generate a new pitch contour P rec ;

[0014] The adversarial speaker classifier group classifies and identifies speakers based on the speech component features of timbre and the combined speech component features of rhythm, content, and pitch;

[0015] The mutual information parameterized network approximates the true conditional distribution between content and rhythm by combining the speech component features of the two.

[0016] (3) Using speech datasets to train the above model through a two-stage multi-task collaborative training method;

[0017] (4) Use pwCCA (projection-weighted Canonical Correlation Analysis) to evaluate the decoupling ability of each speech component of the model, adjust the weights between each decoupling loss based on the evaluation results, and retrain the model for iterative optimization;

[0018] (5) According to the actual speech component conversion requirements, the pitch contours and Mel spectrograms of the source and target speech are input into the trained model to reconstruct a new Mel spectrogram X rec , and then X rec Input into a pre-trained vocoder to generate the desired converted speech.

[0019] Furthermore, in the step (1), the pitch contour of each speech in the data set is first extracted, and then the pitch enhancement processing is performed on each speech, and the Mel spectra of the speech before and after the processing are respectively extracted to obtain the Mel spectra of the original speech and the Mel spectra with enhanced pitch.

[0020] Furthermore, the joint random resampling module first randomly crops the input signal, and then uses a random factor to guide the deformation of each signal segment and splices them together, and the value range of the random factor is 0.5 to 1.5; for the same group of samples, the pitch contour and the Mel spectrum use the same random factor to ensure that the content encoding and the pitch encoding are aligned in the time frame after random resampling.

[0021] Furthermore, the multi-channel encoder group includes a rhythm encoder, a content encoder, a timbre encoder and a pitch encoder, and the output of each encoder is downsampled in the time dimension, wherein the input of the pitch encoder is a randomly resampled pitch contour, and the output is a pitch feature coding matrix f of 64×T size; the input of the content encoder is a randomly resampled Mel spectrum, and the output is a content feature coding matrix c of 16×T size; the input of the rhythm encoder is a pitch-enhanced Mel spectrum, and the output is a rhythm feature coding matrix r of 2×T size; the input of the timbre encoder is the Mel spectrum of the original speech, and the output is a timbre feature coding matrix u of 256×1 size, where T is the number of frames of the speech in the time dimension.

[0022] Furthermore, the adversarial speaker classifier group includes two classifiers: a positive classifier and a negative classifier, wherein the input of the positive classifier is u, the input of the negative classifier is a tensor obtained by concatenating c, r, and f, and the input end of the negative classifier is additionally provided with a gradient reversal layer (GRL).

[0023] Furthermore, in the two-stage multi-task collaborative training method, the Mel-spectrogram reconstruction, pitch contour reconstruction, and adversarial speaker classification initiated in the first stage are used to preliminarily train the decoupling ability of the model's speech components, and the content feature reconstruction and content-rhythm feature mutual information variational upper bound minimization initiated in the second stage further optimize the model's decoupling performance on the basis of the first stage and improve the model's convergence efficiency; the objective function used in the training method is obtained by the weighted sum of six groups of loss functions L1 to L6, and each time the mutual information variational upper bound minimization training task is performed, the mutual information parameterized network therein is subjected to five maximum likelihood trainings.

[0024] Furthermore, the loss functions L1 and L2 are reconstruction loss functions of the Mel spectrum and pitch contour, respectively, and the specific expressions are as follows:

[0025]

[0026] Where: X is the Mel spectrum of the original speech, P is the pitch contour of the original speech, and E() is the expected function.

[0027] Furthermore, the loss functions L3 and L4 are the classification loss functions of the positive classifier and the negative classifier, respectively, and the specific expressions are as follows:

[0028]

[0029] Where: I() is an indicator function, i.e., it takes 1 when the relationship in the brackets is true and 0 when it is false; p k The probability value of the positive classifier or negative classifier output prediction result belonging to the kth class, K is the number of speakers in the data set, Num true is the real speaker number corresponding to the input sample.

[0030] Furthermore, the loss function L5 is a reconstruction loss function of content features, and its expression is as follows:

[0031]

[0032] Where: c rec Mel Score X rec The content feature encoding matrix is ​​input into the content encoder to be reconstructed, and E() is the expected function.

[0033] Furthermore, the loss function L6 is a loss function that minimizes the upper bound of the content-rhythm feature mutual information variation, and its expression is as follows:

[0034]

[0035] Where: N is the number of samples in a batch, i and j are sample index numbers, q θ (c i | ri) is the real distribution p(c i |r i ) The mutual information variational approximate distribution obtained, the true distribution p(c i |r i ) indicates that given r i Under the condition of c i The conditional probability, r i is the rhythm feature encoding matrix of the i-th sample, c i is the content feature encoding matrix of the i-th sample.

[0036] Canonical Correlation Analysis (CCA) is a statistical technique used to measure the correlation between two consecutive-valued random vectors. Mathematically, CCA takes a set of n vector pairs {(x1, y1), ..., (X n ,y n )}, sample from random vectors of dimensions d1 and d2 respectively; let is a matrix of sample column vectors, then CCA aims to identify k = min (d1, d2) on the linear projection W i ,s i , satisfying the following equation:

[0037]

[0038] where any projection is uncorrelated with other projections acting on the same view. The CCA result under trivial conditions is the average of all correlation scores ρ∈[0, 1]; however, pwCCA uses the percentage of projections in different directions as weights to sum the correlation scores, which enables pwCCA to produce more robust results against spurious correlations in the data; the introduction of weights leads to asymmetry, so we use the mean of pwCCA(X, Y) and pwCCA(Y, X).

[0039] Furthermore, in order to evaluate the disentanglement capability between features, in step (4), pwCCA is first calculated for each of the three temporal encoders (rhythm encoder, content encoder, and pitch encoder). i The feature independence of pwCCA and the other two encoders E j and E k The average value is used to characterize, specifically:

[0040] CCAind(E i )=[CCA(E i , E j )+CCA(E i , E k )] / 2

[0041] Where: CCAind(E i ) is the encoder E i The feature independence evaluation index, CCA(E i ,E j ) is the encoder E i With E j pwCCA score between i ,E k ) is the encoder E i With E kThe pwCCA scores between,i,j,k∈[f,c,r], f,c,r correspond to pitch, content, and rhythm;

[0042] According to the above evaluation results, the weights of loss functions L2 to L6 are reset, and the model is retrained for iterative optimization.

[0043] Based on the above technical solution, the present invention has the following beneficial technical effects:

[0044] 1. The present invention introduces a priori understanding of rhythm and pitch information through two data enhancement methods, which helps the encoder group to better prevent interference from rhythm and pitch information when extracting features.

[0045] 2. The present invention fully decouples and extracts the timbre, pitch, content and rhythm information in the speech through the bottleneck encoder group, rather than only disentangling the timbre information in the speech, which greatly increases the degree of freedom of speech conversion.

[0046] 3. The present invention further enhances the decoupling performance of the model through two-stage training.

[0047] 4. The present invention accelerates the convergence of the model and improves the model performance through a two-stage training setting.

[0048] 5. The present invention resets the weighted value of the loss function by evaluating the decoupling ability of the model through pwCCA, thereby optimizing the decoupling ability of the corresponding speech components in the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 The figure is a flow chart of the speech conversion method of the present invention.

[0050] Figure 2 It is a schematic diagram of the structure of the multi-component speech conversion model based on enhanced decoupling in the present invention.

[0051] Figure 3 The figure is a schematic diagram of the process of voice conversion using the model in the present invention.

[0052] Figure 4 This is a spectrogram example of the conversion effect of the speech conversion model of the present invention. The letters in the figure represent different conversion types, that is, U, R, and F represent timbre, rhythm, and pitch respectively. DETAILED DESCRIPTION

[0053] In order to describe the present invention more specifically, the technical solution of the present invention is described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0054] like Figure 1 As shown, the multi-component speech conversion method based on enhanced decoupling and iterative optimization of the present invention comprises the following steps:

[0055] (1) Dataset preprocessing.

[0056] The speech data in this embodiment uses the VCTK data set, in which speech data of 100 speakers are selected for training and speech data of 6 speakers are selected for testing.

[0057] First, the speech data is pitch enhanced to selectively eliminate the pitch information flowing into the encoder. Specifically, the fundamental frequency f0 is extracted using the WORLD vocoder, the fundamental frequency average of all voiced frames in the fundamental frequency is calculated, and the average is used to overwrite all voiced frames in the original fundamental frequency to destroy the pitch information in the input speech (to assist the content encoder and the rhythm encoder to better extract the content and rhythm information), and then the WORLD vocoder is used to generate pitch-enhanced speech from the modified fundamental frequency.

[0058] Then the pitch contour of the speech data is extracted using the WORLD vocoder, and the Mel spectrogram is extracted from the speech data before and after pitch enhancement.

[0059] (2) Random initialization such as Figure 2 The multi-component speech conversion model shown includes: a joint random resampling module, a multi-channel encoder group (rhythm encoder, content encoder, timbre encoder and pitch encoder), a Mel spectrum decoder, a pitch contour decoder, an adversarial speaker classifier group (positive classifier, negative classifier with gradient reversal layer), and a mutual information parameterized network.

[0060] (3) Joint random resampling of the pitch contour and the pitch-enhanced Mel-spectrogram in the time dimension.

[0061] The joint random resampling module first randomly crops the input signal, and then generates a random number factor to guide each signal segment to deform and splice together. The value range of the random factor is 0.5 (compressed to half of the original) to 1.5 (expanded to 1.5 times the original), which is used to selectively eliminate the rhythm information in the input speech (to assist the pitch encoder and content encoder to better extract pitch and content information). Joint random resampling means that for the same set of samples, the pitch contour and the Mel spectrum use the same random factor, so that the time frame alignment of content encoding and pitch encoding can be achieved after random resampling.

[0062] (4) Use a multi-channel encoder group with a bottleneck to extract speech component features.

[0063] The multi-channel encoder group includes a rhythm encoder, a content encoder, a timbre encoder and a pitch encoder. The bottleneck means that the output information of the encoder is limited. In this embodiment, the following two bottleneck operations are specifically used:

[0064] ① Control the feature dimensions of the encoder output. In this embodiment, the specific feature dimensions used are: content feature dimension is 16, rhythm feature dimension is 2, pitch feature dimension is 64, and timbre feature dimension is 256.

[0065] ② Downsample the output of the encoder in the time dimension. In this embodiment, the downsampling parameter used is 8.

[0066] The input of the pitch encoder is the randomly resampled pitch contour, the input of the content encoder is the pitch-enhanced and randomly resampled Mel spectrogram, the input of the rhythm encoder is the pitch-enhanced Mel spectrogram, and the input of the timbre encoder is the original Mel spectrogram. The output f of the pitch encoder has a dimension of 64×T, the output u of the timbre encoder has a dimension of 256×1, the output c of the content encoder has a dimension of 16×T, and the output r of the rhythm encoder has a dimension of 2×T, where T is the number of frames in the time dimension.

[0067] In this embodiment, the content encoder consists of three u The rhythm encoder consists of a one-dimensional convolutional layer with relu activation, a group normalization layer, and two bidirectional LSTM layers; the pitch encoder consists of three re l u The activated one-dimensional convolution layer, group normalization layer, and bidirectional LSTM layer are connected; the timbre encoder consists of 8 rel u Activated one-dimensional convolutional layer, 6 with rel u Activation and residual connection of the 1D convolutional layer, pooling layer, 12 with residual connection and rel u The activated fully connected layer is composed of

[0068] (5) The decoder combines the speech component features to generate the target Mel-spectrograph.

[0069] First, the timbre code u is copied in the time dimension to become a 256×T tensor, and then the copied u is concatenated with f, c, and r to become a 338×T tensor. The Mel spectrum decoder inputs this tensor to generate the target Mel spectrum X rec , the corresponding reconstruction loss function is:

[0070]

[0071] Where: X is the Mel spectrum of the original speech, and E() is the expected function.

[0072] In this embodiment, the Mel-spectrogram decoder is composed of a bidirectional LSTM layer, a linear layer, and five one-dimensional convolutional layers with tanh activation.

[0073] (6) Pitch contour reconstruction and adversarial speaker classifier group.

[0074] Pitch contour reconstruction uses pitch and rhythm information to reconstruct the pitch contour, respectively improving the ability of the pitch encoder and rhythm encoder to extract pitch information and rhythm information. The encoder output f and r are spliced ​​together in the time dimension to generate a 66×T dimensional tensor, which is used as input to the pitch contour decoder to generate the reconstructed pitch contour P rec , the corresponding reconstruction loss function is:

[0075]

[0076] Where: P is the pitch contour of the original speech.

[0077] In this embodiment, the pitch contour decoder is composed of a bidirectional LSTM layer connected with a linear layer with sigmoid activation.

[0078] The adversarial speaker classifier group includes a positive classifier and a negative classifier with a gradient reversal layer, which are used to assist the timbre encoder to better extract timbre information and assist the pitch, content, and rhythm encoders to delete timbre information; the positive classifier receives u as input and predicts and classifies the speaker label; the negative classifier has an additional gradient reversal layer, which receives an 82×T-dimensional tensor concatenated with f, c, and r as input and predicts and classifies the speaker label. The classification loss functions of the two are:

[0079]

[0080] Where: I() is an indicator function, that is, the value in the brackets is 1 when it is true and 0 when it is false; p k The probability value of the kth class in the classifier output prediction result, K is the number of speakers, Num true The real speaker number corresponding to the sample.

[0081] The gradient reversal layer GRL is used to multiply the gradient passed to GRL by a negative number in the gradient backpropagation step of network training, so that the training objectives of the network before and after GRL are opposite; in this embodiment, the classifier is composed of 2 fully connected layers with relu activation and a softmax activation layer.

[0082] (7) Content coding reconstruction and content coding-rhythm coding mutual information minimization training in the second stage of training.

[0083] Content feature reconstruction is used to improve the performance of the content encoder and stabilize the training of mutual information minimization. The Mel spectrum X generated in step (5) is rec Passed to the content encoder to generate the reconstructed content encoding c rec, the corresponding reconstruction loss function is:

[0084]

[0085] The minimization of the mutual information between content features and rhythm features is used to enhance the mutual decoupling performance between content information and rhythm information. The variational upper bound of the minimized mutual information is used to achieve the minimization training of the mutual information. The variational upper bound of the mutual information used is:

[0086]

[0087] Where: N is the number of samples in a batch, i, j are sample indexes, q θ (c i |r i ) is the real distribution p(c i |r i ) is the variational approximation of the mutual information obtained.

[0088] In this embodiment, the mutual information parameterized network is composed of 4 fully connected layers with relu activation. In the mutual information upper bound minimization training, the parameterized network is trained simultaneously using the maximum likelihood. The training of the above two losses is performed alternately. After each mutual information upper bound minimization training, the maximum likelihood training of the parameterized network is performed five times.

[0089] (8) The weighted values ​​of the loss function are reset by evaluating the decoupling ability of the model through pwCCA, thereby optimizing the decoupling ability of the corresponding speech components in the model.

[0090] In this embodiment, in order to evaluate the disentanglement capability between features, we calculate pwCCA between two of the three temporal encoders. i The feature independence of j and E k The average value is represented by:

[0091] CCAind(E i )=[CCA(E i ,E j )+CCA(E i ,E k )] / 2

[0092] Among them: i,j,k∈[f,c,r] and are different from each other.

[0093] According to the evaluation results, the weighted values ​​of the loss functions L2 to L6 are reset, and the model is retrained to iteratively optimize it.

[0094] Combining the above steps (3) to (8) yields a trained multi-component speech conversion model.

[0095] (9) Use the trained multi-component speech conversion model to perform speech component conversion.

[0096] like Figure 3 As shown, firstly, the source speech and the target speech are prepared, and the Mel spectrograms and pitch contours of the source speech and the target speech are extracted.

[0097] Then, according to the actual speech component conversion requirements, the corresponding Mel spectrum or pitch contour is sent to the encoder of the corresponding component. For example, if you want to convert the rhythm, the Mel spectrum of the target speech is sent to the rhythm encoder, and other encoders receive the pitch contour, Mel spectrum and other information of the source speech; if you want to convert the timbre and pitch at the same time, the Mel spectrum of the target speech is input into the timbre encoder, and the pitch contour of the target speech is input into the pitch encoder, and the other encoders receive the pitch contour, Mel spectrum and other information from the source speech.

[0098] The feature codes output from each encoder are concatenated together according to the method in step (5) and input into the decoder to reconstruct the Mel spectrogram of the converted speech.

[0099] Finally, in this embodiment, the Wavenet vocoder pre-trained on the VCTK dataset is used to input the Mel spectrogram of the converted speech, and finally the converted speech we need is generated. The Wavenet vocoder is a vocoder commonly used in speech generation. It uses dilated causal convolution to expand the receptive field of the convolution, and then predicts each sampling point through global information.

[0100] Verification Example

[0101] We experimentally compare the mean opinion scores (MOS) of speech naturalness of the audio generated by the speech conversion model of the present invention with the SRDvc model. The MOS score of speech naturalness is a commonly used subjective evaluation method for evaluating speech conversion systems. Specifically, the subjects give discrete numerical scores on the naturalness of randomly sorted audio, ranging from 1 (bad) to 5 (excellent); SRDvc [Yang Sicheng, Tantrawenith Methawee, Zhuang Haolin Wu Zhiyong Sun Aolan Wang Jianzong Cheng Ning Tang Huaizhen Zhao Xintao Wang Jie and Meng Helen, "Speech Representation Disentanglement with Adversarial Mutual Information Learning for One-shot Voice Conversion," 10.48550 / arXiv.2208.08757] is one of the best multi-component speech conversion systems currently.

[0102] Specifically, we performed single-component conversion of the rhythm, pitch, and timbre components of speech on parallel sentence pairs, and 15 evaluators with good English skills conducted subjective scoring evaluation on the randomly shuffled system-generated speech. The evaluation results are shown in Table 1. It can be seen from the table that in the conversion of various speech components, the MOS score of the speech conversion model of the present invention is slightly higher than that of SRDvc.

[0103] Table 1

[0104] System\Conversion Type Rhythm pitch timbre Model of the present invention 2.34 3.77 3.62 SRDVC Model 2.25 3.45 3.58

[0105] Figure 4 This is a spectrogram sample of the conversion effect of the speech conversion model of the present invention. It can be seen from the figure that for each conversion, except for the converted component becoming similar to the target audio, the remaining components of the audio generated remain consistent with the source audio.

[0106] When evaluating multi-component speech conversion models, MOS and ASR models are usually used to evaluate the naturalness and clarity of speech. On the other hand, the conversion rate is used to consider the decoupling ability between various features of the model. However, this indicator relies on the subjective judgment of the evaluator, and the scale of the evaluation sample is difficult to expand. Therefore, we try to introduce new objective indicators from statistics to quantify it.

[0107] In order to evaluate the feature decoupling ability of the model, we first introduced an objective evaluation index from a statistical perspective. CCA is a statistical technique used to measure the correlation between two continuous-valued random vectors, so it is widely used as a similarity measure to compare the relationship between internal or output features of neural network models. Mathematically, CCA takes a set of n vector pairs {(x1,y1),…,(x n ,y n )}, sample from random vectors of dimensions d1 and d2 respectively; let is a matrix of sample column vectors, then CCA aims to identify k = min (d1, d2) on the linear projection w i, s i , satisfying the following equation:

[0108]

[0109] where any projection is uncorrelated with other projections acting on the same view. The CCA result under trivial conditions is the average of all correlation scores ρ∈[0,1]; however, pwCCA uses the percentage of projections in different directions as weights to sum the correlation scores, which enables pwCCA to produce more robust results against spurious correlations in the data; the introduction of weights leads to asymmetry, so we use the mean of pwCCA(X,Y) and pwCCA(Y,X).

[0110] In order to evaluate the disentanglement ability between features, we calculate pwCCA between the three temporal encoders. i The feature independence of j and E k The average value is represented by:

[0111] CCAind(E i )=[CCA(E i ,E j )+CCA(E i ,E k )] / 2

[0112] Among them: i,j,k∈[f,c,r] and are different from each other.

[0113] Since the present invention is calculated directly on the features without generating audio, the expansion of the test sample set is very convenient, thereby providing more robust results, as shown in Table 2:

[0114] Table 2

[0115] pC <![CDATA[E r ]]> <![CDATA[E f ]]> <![CDATA[E c ]]> The present invention 0.52 0.32 0.45 SRDVC Model 0.58 0.45 0.50

[0116] As a supplementary evidence for the introduction of statistical indicators, subjectively, we use conversion rate to evaluate the decoupling ability. For each conversion, we generate conversion speech on parallel sentences. Participants randomly listen to the source speech and the target speech and judge which one the generated speech component is more similar to. The conversion rate of the component is the ratio of samples judged to be more similar to the target sample to the total samples:

[0117] CR i =target i / total i

[0118] This indicator subjectively reflects the decoupling ability of the model, as shown in Table 3:

[0119] Table 3

[0120]

[0121] The above subjective and objective indicators show that the model of the present invention has better ability to extract inter-feature disentanglement.

[0122] The above description of the embodiments is to facilitate the understanding and application of the present invention by those skilled in the art. It is obvious that those skilled in the art can easily make various modifications to the above embodiments and apply the general principles described herein to other embodiments without creative work. Therefore, the present invention is not limited to the above embodiments. Improvements and modifications made by those skilled in the art to the present invention based on the disclosure of the present invention should be within the protection scope of the present invention.

Claims

1. A multi-component speech conversion method based on enhanced decoupling and iterative optimization, comprising the following steps: (1) Obtain a speech dataset, perform pitch enhancement processing on each speech, and extract the pitch contour and Mel spectrogram; (2) Construct a multi-component speech conversion model, which includes: Joint random resampling module, used to perform joint random resampling of pitch contour and pitch-enhanced Mel spectrogram in the time dimension; A multi-channel encoder group with information bottleneck extracts speech component features corresponding to content, rhythm, timbre, and pitch based on pitch contours and Mel spectrograms; Mel spectrogram decoder, combining all speech component features to reconstruct a new Mel spectrogram X rec ; Auxiliary pitch contour decoder, which reconstructs the speech component features of pitch and rhythm to generate a new pitch contour P rec ; The adversarial speaker classifier group classifies and identifies speakers based on the speech component features of timbre and the combined speech component features of rhythm, content, and pitch; The mutual information parameterized network approximates the true conditional distribution between content and rhythm by combining the speech component features of the two. (3) Using speech datasets to train the above model through a two-stage multi-task collaborative training method; (4) Use pwCCA to evaluate the decoupling ability of each speech component of the model, adjust the weights between the decoupling losses based on the evaluation results, and retrain the model for iterative optimization; (5) According to the actual speech component conversion requirements, the pitch contours and Mel spectrograms of the source and target speech are input into the trained model to reconstruct a new Mel spectrogram X rec , and then X rec Input into a pre-trained vocoder to generate the desired converted speech.

2. The multi-component speech conversion method according to claim 1, characterized in that: In the step (1), the pitch contour of each speech in the data set is firstly extracted, and then the pitch enhancement processing is performed on each speech, and the Mel spectra of the speech before and after the processing are respectively extracted to obtain the Mel spectra of the original speech and the Mel spectra with enhanced pitch.

3. The multi-component speech conversion method according to claim 1, characterized in that: The joint random resampling module first randomly crops the input signal, and then uses a random factor to guide the deformation of each signal segment and splices them together. The value range of the random factor is 0.5 to 1.

5. For the same group of samples, the pitch contour and the Mel spectrum use the same random factor to ensure that the content encoding and the pitch encoding are aligned in the time frame after random resampling.

4. The multi-component speech conversion method according to claim 1, characterized in that: The multi-channel encoder group includes a rhythm encoder, a content encoder, a timbre encoder and a pitch encoder, and the output of each encoder is downsampled in the time dimension, wherein the input of the pitch encoder is a pitch contour after random resampling, and the output is a pitch feature coding matrix f of 64×T size; the input of the content encoder is a Mel spectrum after random resampling, and the output is a content feature coding matrix c of 16×T size; the input of the rhythm encoder is a Mel spectrum with pitch enhancement, and the output is a rhythm feature coding matrix r of 2×T size; the input of the timbre encoder is the Mel spectrum of the original speech, and the output is a timbre feature coding matrix u of 256×1 size, where T is the number of frames of the speech in the time dimension.

5. The multi-component speech conversion method according to claim 4, characterized in that: The adversarial speaker classifier group includes two classifiers: a positive classifier and a negative classifier, wherein the input of the positive classifier is u, the input of the negative classifier is a tensor obtained by concatenating c, r, and f, and the input end of the negative classifier is additionally provided with a gradient reversal layer.

6. The multi-component speech conversion method according to claim 4, characterized in that: In the two-stage multi-task collaborative training method, the Mel-spectrogram reconstruction, pitch contour reconstruction, and adversarial speaker classification started in the first stage are used to preliminarily train the decoupling ability of the model's speech components, and the content feature reconstruction and content-rhythm feature mutual information variational upper bound minimization started in the second stage further optimize the model's decoupling performance on the basis of the first stage and improve the model's convergence efficiency; the objective function used in the training method is obtained by the weighted sum of six groups of loss functions L1 to L6, and each time the mutual information variational upper bound minimization training task is performed, the mutual information parameterized network therein is subjected to five maximum likelihood trainings.

7. The multi-component speech conversion method according to claim 6, characterized in that: The loss functions L1 and L2 are the reconstruction loss functions of Mel spectrogram and pitch contour respectively, and the specific expressions are as follows: Where: X is the Mel spectrum of the original speech, P is the pitch contour of the original speech, and E() is the expected function.

8. The multi-component speech conversion method according to claim 6, characterized in that: The loss functions L3 and L4 are the classification loss functions of the positive classifier and the negative classifier respectively, and the specific expressions are as follows: Where: I() is an indicator function, i.e., it takes 1 when the relationship in the brackets is true and 0 when it is false; p k The probability value of the positive classifier or negative classifier output prediction result belonging to the kth class, K is the number of speakers in the data set, Num true is the real speaker number corresponding to the input sample.

9. The multi-component speech conversion method according to claim 6, characterized in that: The loss function L5 is a reconstruction loss function of content features, and its expression is as follows: Where: c rec Mel Score X rec Input to the content encoder to reconstruct the generated content feature encoding matrix, E() is the expected function; The loss function L6 is a loss function that minimizes the upper bound of the content-rhythm feature mutual information variation, and its expression is as follows: Where: N is the number of samples in a batch, i and j are sample index numbers, q θ (c i |r i ) is the real distribution p(c i |r i ) The mutual information variational approximate distribution obtained, the true distribution p(c i |r i ) indicates that given r i Under the condition c i The conditional probability, r i is the rhythm feature encoding matrix of the i-th sample, c i is the content feature encoding matrix of the i-th sample.

10. The multi-component speech conversion method according to claim 6, characterized in that: In order to evaluate the detangling capability between features, in step (4), pwCCA calculation is first performed between two of the three temporal encoders. For any encoder E i The feature independence of pwCCA and the other two encoders E j and E k The average value is used to characterize, specifically: CCInd(E i )=[CCA(E i ,It j )+CCA(E i ,It k )] / 2 Where: CCAind(E i ) is the encoder E i The feature independence evaluation index, CCA(E i ,E j ) is the encoder E i With E j pwCCA score between i ,E k ) is the encoder E i With E k The pwCCA scores between,i,j,k∈[f,c,r], f,c,r correspond to pitch, content, and rhythm; According to the above evaluation results, the weights of loss functions L2 to L6 are reset, and the model is retrained for iterative optimization.

Images