An iteratively optimized multi-component speech conversion method based on enhanced decoupling

Abstract

This invention discloses an iteratively optimized multi-component speech conversion method based on enhanced decoupling. This method utilizes a multi-channel encoder group with an information bottleneck to extract rhythm, content, and timbre information from speech, as well as pitch information from the pitch contour. A decoder is then used to generate speech in conjunction with the encoder group. Random resampling and pitch enhancement are used to selectively eliminate rhythm or pitch information flowing into the encoder. The model's speech disentanglement capability is then enhanced through a multi-task training method involving collaborative pitch contour reconstruction, an adversarial speaker classifier group, content feature reconstruction, and content-rhythm feature mutual information minimization. Furthermore, the invention simplifies the training process and improves the model's convergence efficiency through a two-stage approach. This approach aims to achieve high-quality multi-component speech conversion, addressing the problem that traditional speech conversion methods can only decouple timbre while other speech components remain entangled.

Description

Technical Field

[0001] The present invention belongs to the technical field of speech processing, and in particular relates to an iteratively optimized multi-component speech conversion method based on enhanced decoupling. Background Art

[0002] Human language is a treasure trove of information, comprised of four fundamental components: content, pitch, rhythm, and timbre. Content conveys semantic meaning, pitch and rhythm shape the emotional tone of speech through the frequency and length of syllables, and timbre serves as a marker of speaker identity. Many speech applications are designed to target these specific components: automatic speech recognition programs aim to transcribe spoken content into text, automatic speaker verification extracts timbre, and singing voice synthesis programs manipulate pitch and rhythm. However, the challenge of disentangling these intertwined components has sparked widespread interest and effort.

[0003] Traditional speech conversion technology attempts to separate the speaker's information from the source speech while retaining high-level factors such as content and emotion, so that the speech remains consistent with the timbre of the target speaker. Previous studies have achieved better results by incorporating prior settings into the model structure. For example, the literature [Lifa Sun, Kun Li, Hao Wang, Shiyin Kang, and Helen M. Meng, “Phonetic posteriorgrams for many-to-one voice conversion without parallel data training,” in IEEE International Conference on Multimedia and Expo, ICME 2016, Seattle, WA, USA, July 11-15, 2016.2016, pp.1–6, IEEE Computer Society] proposed using the phonetic posteriorgrams (PPGs) of the pre-trained speech recognition model as speaker-independent information to decouple timbre; another example is the literature [Xintao Zhao, Feng Liu, Changhe Song, Zhiyong Wu, Shiyin Kang, Deyi Tuo, and Helen Meng, “Disentangling content and fine-grained prosody information via hybrid ASR bottleneck features for voice conversion,” in Proc. ICASSP. 2022, pp. 7022–7026, IEEE] proposed using a bottleneck encoder (BNE) to forcibly decouple timbre by controlling the representational degrees of freedom of the feature latent space. However, while these traditional speech conversion technologies have achieved certain results, they are still plagued by intertwined factors such as pitch and rhythm. This can cause the generated converted audio to exhibit rhythmic mutations, resulting in poor conversion results. Furthermore, existing technologies can only decouple and edit the timbre information in the speech, making it difficult to effectively control the pitch and rhythm information in the speech.

[0004] To address these issues, the paper [Kaizhi Qian, Yang Zhang, Shiyu Chang, Mark Hasegawa-Johnson, and David Cox, “Unsupervised speech decomposition via triple information bottleneck,” in Proceedings of the 37th International Conference on Machine Learning, 2020, pp. 7836–7846] proposed SpeechSplit, which pioneered the decoupling of multiple components involving speech timbre, pitch, rhythm, and content. It utilizes a multi-channel bottleneck encoder group to extract content, timbre, and rhythmic information from the Mel-spectrogram, as well as pitch information from the pitch contour. It also uses random resampling to destroy the rhythmic information in the Mel-spectrogram, thereby improving the encoder's ability to extract the corresponding information. However, SpeechSplit's decoupling effect relies heavily on the design of the bottleneck parameters in the bottleneck encoder, which require fine-tuning for different scenarios and data in different training sets. This consumes a lot of manpower and computing resources in actual use.

[0005] Therefore, we urgently need a robust, high-quality speech conversion method that can fully decouple the various components of speech, in order to address the shortcomings of traditional methods in terms of pitch, rhythm, etc., while improving the robustness of the decoupling capability under different data distributions, thereby achieving better multi-component speech conversion effects. Specifically, the present invention proposes a novel multi-component speech conversion system that can perform high-quality detangling and conversion of the rhythm, pitch, and timbre in speech with a single sample; at the same time, a two-stage training strategy is designed to accelerate the convergence of the model while improving the model effect; in addition, the present invention uses pwCCA to evaluate the decoupling ability of each component of the model. Based on the evaluation results, the weighted values ​​of the loss function are reset, and the model is retrained for iterative optimization. Summary of the Invention

[0006] In view of the above, the present invention provides an iteratively optimized multi-component speech conversion method based on enhanced decoupling. By using a priori random resampling, pitch enhancement and two-stage decoupling training techniques, it solves the problems that the traditional timbre conversion model can only decouple timbre while other speech components are still entangled, resulting in the generated speech rhythm jumping and uncontrollable problems. The model enables high-quality arbitrary conversion of timbre, pitch and rhythm components in speech.

[0007] A multi-component speech conversion method based on enhanced decoupling and iterative optimization includes the following steps:

[0008] (1) Obtain a speech dataset, perform pitch enhancement on each speech, and extract the pitch contour and Mel spectrogram;

[0009] (2) Constructing a multi-component speech conversion model, which includes:

[0010] Joint random resampling module, used to perform joint random resampling of the pitch contour and the pitch-enhanced Mel-spectrogram in the time dimension;

[0011] A multi-channel encoder group with an information bottleneck extracts speech component features corresponding to content, rhythm, timbre, and pitch based on pitch contours and Mel spectrograms;

[0012] Mel spectrum decoder, which combines all speech component features to reconstruct a new Mel spectrum X rec ;

[0013] Auxiliary pitch contour decoder, which jointly reconstructs the speech component features of pitch and rhythm to generate a new pitch contour P rec ;

[0014] The adversarial speaker classifier group classifies and identifies speakers based on the voice component characteristics of timbre and the combined voice component characteristics of rhythm, content, and pitch;

[0015] Mutual information parameterized network, which approximates the true conditional distribution between content and rhythm by combining speech component features;

[0016] (3) Using speech datasets to train the above model through a two-stage multi-task collaborative training method;

[0017] (4) Use pwCCA (projection-weighted Canonical Correlation Analysis) to evaluate the decoupling ability of each speech component of the model. Based on the evaluation results, adjust the weights between the various decoupling losses and retrain the model for iterative optimization.

[0018] (5) According to the actual speech component conversion requirements, the pitch contours and Mel spectra of the source and target speech are input into the trained model to reconstruct the new Mel spectra X rec , and then X rec Input into a pre-trained vocoder to generate the required converted speech.

[0019] Furthermore, in step (1), the pitch contour of each speech in the data set is first extracted, and then each speech is pitch enhanced, and the mel spectra of the speech before and after the processing are extracted respectively to obtain the mel spectra of the original speech and the mel spectra of the pitch enhanced speech.

[0020] Furthermore, the joint random resampling module first randomly crops the input signal, and then uses a random factor to guide the deformation of each signal segment and splices them together. The value range of the random factor is 0.5 to 1.5. For the same group of samples, the pitch contour and the Mel spectrum use the same random factor to ensure that the content encoding and the pitch encoding are aligned in the time frame after random resampling.

[0021] Furthermore, the multi-channel encoder group includes a rhythm encoder, a content encoder, a timbre encoder and a pitch encoder, and the output of each encoder is downsampled in the time dimension, wherein the input of the pitch encoder is the pitch contour after random resampling, and the output is a pitch feature coding matrix f of 64×T size; the input of the content encoder is the Mel spectrum after random resampling, and the output is a content feature coding matrix c of 16×T size; the input of the rhythm encoder is the pitch-enhanced Mel spectrum, and the output is a rhythm feature coding matrix r of 2×T size; the input of the timbre encoder is the Mel spectrum of the original speech, and the output is a timbre feature coding matrix u of 256×1 size, where T is the number of frames of the speech in the time dimension.

[0022] Furthermore, the adversarial speaker classifier group includes two classifiers: a positive classifier and a negative classifier, wherein the input of the positive classifier is u, and the input of the negative classifier is a tensor obtained by concatenating c, r, and f, and the input end of the negative classifier is additionally provided with a gradient reversal layer (GRL).

[0023] Furthermore, in the two-stage multi-task collaborative training method, the Mel-spectrogram reconstruction, pitch contour reconstruction, and adversarial speaker classification initiated in the first stage are used to preliminarily train the decoupling ability of the model's speech components, and the content feature reconstruction and content-rhythm feature mutual information variational upper bound minimization initiated in the second stage further optimize the model's decoupling performance on the basis of the first stage and improve the model's convergence efficiency; the objective function adopted by the training method is obtained by the weighted sum of six groups of loss functions L1 to L6, and each time the mutual information variational upper bound minimization training task is performed, the mutual information parameterized network is subjected to five maximum likelihood trainings.

[0024] Furthermore, the loss functions L1 and L2 are reconstruction loss functions of Mel spectrum and pitch contour respectively, and the specific expressions are as follows:

[0025]

[0026] Where: X is the Mel spectrum of the original speech, P is the pitch contour of the original speech, and E() is the expectation function.

[0027] Furthermore, the loss functions L3 and L4 are the classification loss functions of the positive classifier and the negative classifier, respectively, and the specific expressions are as follows:

[0028]

[0029] Where: I() is an indicator function, i.e., it takes 1 when the relationship in the brackets is true and 0 when it is false; k The probability value of the predicted result of the positive or negative classifier output belongs to the kth class, K is the number of speakers in the dataset, Num true The real speaker number corresponding to the input sample.

[0030] Furthermore, the loss function L5 is a reconstruction loss function of content features, and its expression is as follows:

[0031]

[0032] Where: c rec Mel Score X rec Input into the content encoder to reconstruct the generated content feature encoding matrix, E() is the expectation function.

[0033] Furthermore, the loss function L6 is a loss function that minimizes the upper bound of the variational mutual information of content-rhythm features, and its expression is as follows:

[0034]

[0035] Where: N is the number of samples in a batch, i and j are sample index numbers, q θ (c i | ri) is the real distribution p(c i |r i ) The mutual information variation approximate distribution obtained, the true distribution p(c i |r i ) indicates that given r i Under the conditions c i The conditional probability, r i is the rhythm feature encoding matrix of the i-th sample, c i is the content feature encoding matrix of the i-th sample.

[0036] Canonical Correlation Analysis (CCA) is a statistical technique used to measure the correlation between two consecutive valued random vectors. Mathematically, CCA takes a set of n vector pairs {(x1, y1), ..., (X n ,y n )}, sample from random vectors of dimensions d1 and d2 respectively; let is a matrix of sample column vectors, then CCA aims to identify k=min(d1, d2) on the linear projection W i , s i , satisfying the following equation:

[0037]

[0038] In this paper, any projection is uncorrelated with other projections acting on the same view. The CCA result under trivial conditions is the average of all correlation scores ρ∈[0, 1]; however, pwCCA uses the percentage of projections in different directions as weights to sum the correlation scores, which enables pwCCA to produce more robust results to spurious correlations in the data; the introduction of weights leads to asymmetry, so we use the mean of pwCCA(X, Y) and pwCCA(Y, X).

[0039] Furthermore, in order to evaluate the ability of detangling between features, the pwCCA is first calculated for each of the three temporal encoders (rhythm encoder, content encoder, and pitch encoder) in step (4). i The feature independence of pwCCA and the other two encoders E j and E k The average value is used to characterize, specifically:

[0040] CCAind(E i )=[CCA(E i , E j )+CCA(E i , E k )] / 2

[0041] Among them: CCAind(E i ) is the encoder E i The feature independence evaluation index, CCA(E i ,E j ) is the encoder E i With E j The pwCCA score between i ,E k ) is the encoder E i With E kThe pwCCA scores between,i,j,k∈[f,c,r], where f,c,r correspond to pitch, content, and rhythm;

[0042] Based on the above evaluation results, the weights of the loss functions L2 to L6 are reset, and the model is retrained for iterative optimization.

[0043] Based on the above technical solution, the present invention has the following beneficial technical effects:

[0044] 1. The present invention introduces a priori understanding of rhythm and pitch information through two data enhancement methods, which helps the encoder group better prevent interference from rhythm and pitch information when extracting features.

[0045] 2. The present invention fully decouples and extracts the timbre, pitch, content and rhythm information in the speech through the bottleneck encoder group, rather than only detangling the timbre information in the speech, which greatly increases the freedom of speech conversion.

[0046] 3. The present invention further enhances the decoupling performance of the model through two-stage training.

[0047] 4. The present invention accelerates the convergence of the model and improves the model performance through a two-stage training setting.

[0048] 5. The present invention resets the weighted value of the loss function by evaluating the decoupling ability of the model through pwCCA, thereby optimizing the decoupling ability of the corresponding speech components in the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 Schematic diagram of the flow of the voice conversion method of the present invention.

[0050] Figure 2 Schematic diagram of the structure of the multi-component speech conversion model based on enhanced decoupling in the present invention.

[0051] Figure 3 This is a schematic diagram of the process of voice conversion using a model in the present invention.

[0052] Figure 4 This is a spectrogram example of the conversion effect of the speech conversion model of the present invention. The letters in the figure represent different conversion types, that is, U, R, and F represent timbre, rhythm, and pitch respectively. DETAILED DESCRIPTION

[0053] In order to describe the present invention more specifically, the technical solution of the present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0054] like Figure 1 As shown, the multi-component speech conversion method based on enhanced decoupling and iterative optimization of the present invention includes the following steps:

[0055] (1) Dataset preprocessing.

[0056] The speech data in this embodiment uses the VCTK dataset, in which speech data of 100 speakers are selected for training and speech data of 6 speakers are selected for testing.

[0057] First, the speech data is pitch-enhanced to selectively eliminate the pitch information flowing into the encoder. Specifically, the WORLD vocoder is used to extract the fundamental frequency f0, and the average fundamental frequency of all voiced frames in the fundamental frequency is calculated. This average value is overwritten with all voiced frames in the original fundamental frequency to destroy the pitch information in the input speech (to assist the content encoder and rhythm encoder in better extracting content and rhythm information). The WORLD vocoder is then used to generate pitch-enhanced speech from the modified fundamental frequency.

[0058] Then the WORLD vocoder is used to extract the pitch contour of the speech data, and the Mel spectrograms are extracted from the speech data before and after pitch enhancement.

[0059] (2) Random initialization such as Figure 2 The multi-component speech conversion model shown includes: a joint random resampling module, a multi-channel encoder group (rhythm encoder, content encoder, timbre encoder and pitch encoder), a Mel spectrum decoder, a pitch contour decoder, an adversarial speaker classifier group (positive classifier, negative classifier with gradient reversal layer), and a mutual information parameterized network.

[0060] (3) Joint random resampling of the pitch contour and the pitch-enhanced Mel-spectrogram in the time dimension.

[0061] The joint random resampling module first randomly crops the input signal and then generates a random number factor to guide the deformation and splicing of each signal segment. The random factor ranges from 0.5 (compression to half the original value) to 1.5 (expansion to 1.5 times the original value). It is used to selectively eliminate rhythmic information in the input speech (to assist the pitch encoder and content encoder in better extracting pitch and content information). Joint random resampling refers to using the same random factor for the pitch contour and Mel-level spectrum for the same set of samples. This ensures that the time frames of the content encoding and pitch encoding are aligned after random resampling.

[0062] (4) Use a multi-channel encoder group with a bottleneck to extract speech component features.

[0063] The multi-channel encoder group includes a rhythm encoder, a content encoder, a timbre encoder, and a pitch encoder. A bottleneck means that the output information of the encoder is limited. In this embodiment, the following two bottleneck operations are specifically adopted:

[0064] ① Control the feature dimensions output by the encoder. In this embodiment, the specific feature dimensions used are: content feature dimension is 16, rhythm feature dimension is 2, pitch feature dimension is 64, and timbre feature dimension is 256.

[0065] ② Downsample the encoder output in the time dimension. In this embodiment, a downsampling parameter of 8 is used.

[0066] The input to the pitch encoder is a randomly resampled pitch contour, the input to the content encoder is a pitch-enhanced and randomly resampled Mel-spectrogram, the input to the rhythm encoder is the pitch-enhanced Mel-spectrogram, and the input to the timbre encoder is the original Mel-spectrogram. The output f of the pitch encoder has a dimension of 64×T, the output u of the timbre encoder has a dimension of 256×1, the output c of the content encoder has a dimension of 16×T, and the output r of the rhythm encoder has a dimension of 2×T, where T is the number of time frames.

[0067] In this embodiment, the content encoder consists of three u The network consists of a one-dimensional convolution layer with relu activation, a group normalization layer, and two bidirectional LSTM layers; the rhythm encoder consists of a one-dimensional convolution layer with relu activation, a group normalization layer, and a bidirectional LSTM layer; the pitch encoder consists of three layers with re l u The activated one-dimensional convolution layer, group normalization layer, and bidirectional LSTM layer are connected; the timbre encoder consists of 8 rel u Activated one-dimensional convolution layer, 6 with rel u One-dimensional convolutional layer with activation and residual connection, pooling layer, 12 layers with residual connection and rel u The activated fully connected layer is connected.

[0068] (5) The decoder combines the speech component features to generate the target Mel spectrogram.

[0069] First, the timbre code u is copied in the time dimension to become a 256×T tensor. Then the copied u is concatenated with f, c, and r to form a 338×T tensor. The Mel spectrum decoder inputs this tensor to generate the target Mel spectrum X. rec , the corresponding reconstruction loss function is:

[0070]

[0071] Where: X is the Mel spectrum of the original speech, and E() is the expected function.

[0072] In this embodiment, the Mel-spectrogram decoder consists of a bidirectional LSTM layer, a linear layer, and five one-dimensional convolutional layers with tanh activation.

[0073] (6) Pitch contour reconstruction and adversarial speaker classifier group.

[0074] Pitch contour reconstruction uses pitch and rhythm information to reconstruct the pitch contour, improving the ability of the pitch encoder and rhythm encoder to extract pitch information and rhythm information respectively. The encoder output f and r are spliced ​​together in the time dimension to generate a 66×T dimensional tensor, which is used as input to the pitch contour decoder to generate the reconstructed pitch contour P rec , the corresponding reconstruction loss function is:

[0075]

[0076] Where: P is the pitch contour of the original speech.

[0077] In this embodiment, the pitch contour decoder consists of a bidirectional LSTM layer connected with a linear layer with sigmoid activation.

[0078] The adversarial speaker classifier group consists of a positive classifier and a negative classifier with a gradient reversal layer, which are used to assist the timbre encoder in better extracting timbre information and assist the pitch, content, and rhythm encoders in removing timbre information. The positive classifier receives u as input and predicts and classifies the speaker label. The negative classifier has an additional gradient reversal layer and receives an 82×T-dimensional tensor concatenated with f, c, and r as input to predict and classify the speaker label. The classification loss functions of the two are:

[0079]

[0080] Where: I() is the indicator function, that is, if the value in the brackets is true, it takes 1, and if it is false, it takes 0; p k The probability value of the kth category in the classifier output prediction result, K is the number of speakers, Num true The real speaker number corresponding to the sample.

[0081] The gradient reversal layer GRL is used to multiply the gradient input to GRL by a negative number in the gradient backpropagation step of network training, so that the training objectives of the network before and after GRL are opposite; in this embodiment, the classifier is composed of two fully connected layers with relu activation and a softmax activation layer.

[0082] (7) Content coding reconstruction and content coding-rhythm coding mutual information minimization training in the second stage of training.

[0083] Content feature reconstruction is used to improve the performance of the content encoder and to stabilize the training of mutual information minimization. The Mel spectrum X generated in step (5) is converted to rec Passed to the content encoder to generate the reconstructed content encoding c rec, the corresponding reconstruction loss function is:

[0084]

[0085] Minimizing the mutual information between content features and rhythm features is used to enhance the decoupling performance between content information and rhythm information. The variational upper bound of the mutual information is used to achieve the minimum training of mutual information. The variational upper bound of the mutual information used is:

[0086]

[0087] Where: N is the number of samples in a batch, i, j are sample indexes, q θ (c i |r i ) is the real distribution p(c i |r i ) is the variational approximate distribution of the mutual information obtained.

[0088] In this embodiment, the mutual information parameterized network is composed of four fully connected layers with ReLU activation. During the mutual information upper bound minimization training, the parameterized network is trained simultaneously using the maximum likelihood. The training of the above two losses is performed alternately. After each mutual information upper bound minimization training, the maximum likelihood training of the parameterized network is performed five times.

[0089] (8) The weighted values ​​of the loss function are reset by evaluating the decoupling ability of the model through pwCCA, thereby optimizing the decoupling ability of the corresponding speech components in the model.

[0090] In this embodiment, in order to evaluate the ability of detangling between features, we calculate pwCCA between two of the three temporal encoders. i The feature independence of pwCCA and the other two encoders E j and E k The average value is represented by:

[0091] CCAind(E i )=[CCA(E i ,E j )+CCA(E i ,E k )] / 2

[0092] Among them: i,j,k∈[f,c,r] and are different from each other.

[0093] According to the evaluation results, the weighted values ​​of the loss functions L2 to L6 are reset, and the model is retrained for iterative optimization.

[0094] Combining the above steps (3) to (8) yields a trained multi-component speech conversion model.

[0095] (9) Use the trained multi-component speech conversion model to convert speech components.

[0096] like Figure 3 As shown, firstly, the source speech and the target speech are prepared, and the Mel spectrograms and pitch contours of the source speech and the target speech are extracted.

[0097] Then, according to the actual speech component conversion requirements, the corresponding Mel spectrum or pitch contour is sent to the encoder of the corresponding component. For example: if you want to convert the rhythm, the Mel spectrum of the target speech is sent to the rhythm encoder, and other encoders receive the pitch contour, Mel spectrum and other information of the source speech; if you want to convert the timbre and pitch at the same time, the Mel spectrum of the target speech is input into the timbre encoder, and the pitch contour of the target speech is input into the pitch encoder, and the other encoders receive the pitch contour, Mel spectrum and other information from the source speech.

[0098] The feature codes output from each encoder are concatenated together according to the method in step (5) and input into the decoder to reconstruct the Mel spectrogram of the converted speech.

[0099] Finally, this example uses the Wavenet vocoder, pre-trained on the VCTK dataset, to input the mel-spectrogram of the converted speech, ultimately generating the desired converted speech. The Wavenet vocoder is a commonly used vocoder in speech generation. It uses dilated causal convolution to expand the receptive field of the convolution and then predict each sample point based on global information.

[0100] Verification Example

[0101] We experimentally compared the Mean Opinion Scores (MOS) of speech naturalness of the audio generated by our speech conversion model with the SRDvc model. The MOS score for speech naturalness is a commonly used subjective evaluation method for speech conversion systems. Specifically, subjects assign a discrete numerical rating of the naturalness of randomly sorted audio sequences, ranging from 1 (poor) to 5 (excellent). SRDvc [Yang Sicheng, Tantrawenith Methawee, Zhuang Haolin, WuZhiyong, Sun Aolan, Wang Jianzong, Cheng Ning, Tang Huaizhen, Zhao Xintao, Wang Jie, and Meng Helen, “Speech Representation Disentanglement with Adversarial MutualInformation Learning for One-shot Voice Conversion,” 10.48550 / arXiv.2208.08757] is currently one of the most effective multi-component speech conversion systems.

[0102] Specifically, we performed single-component conversion of the rhythm, pitch, and timbre components of speech on parallel sentence pairs. Fifteen evaluators with excellent English skills subjectively scored the randomly shuffled system-generated speech. The evaluation results are shown in Table 1. As can be seen from the table, the speech conversion model of the present invention achieves slightly higher MOS scores than SRDvc in converting various speech components.

[0103] Table 1

[0104] System\Conversion Type Rhythm pitch timbre Model of the present invention 2.34 3.77 3.62 SRDVC model 2.25 3.45 3.58

[0105] Figure 4 This is a spectrogram example of the conversion effect of the speech conversion model of the present invention. It can be seen from the figure that for each conversion, except for the converted components that become similar to the target audio, the remaining components of the audio generated remain consistent with the source audio.

[0106] MOS and ASR models are commonly used to evaluate the naturalness and clarity of speech when evaluating multi-component speech conversion models. Conversion rate is also used to assess the decoupling ability between various model features. However, these metrics rely on the subjective judgment of the evaluator, and the sample size is difficult to expand. Therefore, we attempt to introduce new objective statistical metrics to quantify these metrics.

[0107] In order to evaluate the feature decoupling ability of the model, we first introduced an objective evaluation metric from a statistical perspective. CCA is a statistical technique used to measure the correlation between two continuous-valued random vectors. Therefore, it is widely used as a similarity metric to compare the relationship between internal or output features of neural network models. Mathematically, CCA calculates a set of n vector pairs {(x1,y1),…,(x n ,y n )}, sample from random vectors of dimensions d1 and d2 respectively; let is a matrix of sample column vectors, then CCA aims to identify k=min(d1,d2) on the linear projection w i, s i , satisfying the following equation:

[0108]

[0109] In this paper, any projection is uncorrelated with other projections acting on the same view. The CCA result under trivial conditions is the average of all correlation scores ρ∈[0,1]; however, pwCCA uses the percentage of projections in different directions as weights to sum the correlation scores, which enables pwCCA to produce more robust results to spurious correlations in the data; the introduction of weights leads to asymmetry, so we use the mean of pwCCA(X,Y) and pwCCA(Y,X).

[0110] In order to evaluate the disentanglement ability between features, we calculate pwCCA between two of the three temporal encoders. i The feature independence of pwCCA and the other two encoders E j and E k The average value is represented by:

[0111] CCAind(E i )=[CCA(E i ,E j )+CCA(E i ,E k )] / 2

[0112] Among them: i,j,k∈[f,c,r] and are different from each other.

[0113] Since the present invention is calculated directly on the features without generating audio, the expansion of the test sample set is very convenient, thereby providing more robust results, as shown in Table 2:

[0114] Table 2

[0115] pwCCA <![CDATA[E r ]]> <![CDATA[E f ]]> <![CDATA[E c ]]> The present invention 0.52 0.32 0.45 SRDVC model 0.58 0.45 0.50

[0116] As a supplementary evidence for the introduction of statistical indicators, we subjectively evaluated the decoupling ability using the conversion rate. For each conversion, we generated the converted speech on parallel sentences. Participants randomly listened to the source and target speech and judged which of the generated speech components was more similar to the source and target speech components. The conversion rate of the component was the ratio of samples judged to be more similar to the target sample to the total number of samples:

[0117] CR i =target i / total i

[0118] This indicator subjectively reflects the decoupling ability of the model, as shown in Table 3:

[0119] Table 3

[0120]

[0121] The above subjective and objective indicators show that the model of the present invention has better ability to extract inter-feature disentanglement.

[0122] The above description of the embodiments is intended to facilitate understanding and application of the present invention by those skilled in the art. It is apparent that those skilled in the art can readily make various modifications to the above embodiments and apply the general principles described herein to other embodiments without requiring creative effort. Therefore, the present invention is not limited to the above embodiments. Any improvements or modifications made by those skilled in the art based on the disclosure of the present invention should fall within the scope of protection of the present invention.

Claims

1. A multi-component speech conversion method based on enhanced decoupling and iterative optimization, comprising the following steps: (1) Obtain a speech dataset, perform pitch enhancement on each speech, and extract the pitch contour and Mel spectrogram; (2) Constructing a multi-component speech conversion model, which includes: Joint random resampling module, used to perform joint random resampling of the pitch contour and the pitch-enhanced Mel-spectrogram in the time dimension; A multi-channel encoder group with an information bottleneck extracts speech component features corresponding to content, rhythm, timbre, and pitch based on pitch contours and Mel spectrograms; Mel spectrum decoder, which combines all speech component features to reconstruct a new Mel spectrum X rec ; Auxiliary pitch contour decoder, which jointly reconstructs the speech component features of pitch and rhythm to generate a new pitch contour P rec ; The adversarial speaker classifier group classifies and identifies speakers based on the voice component characteristics of timbre and the combined voice component characteristics of rhythm, content, and pitch; Mutual information parameterized network, which approximates the true conditional distribution between content and rhythm by combining speech component features; (3) The above model is trained using a speech dataset through a two-stage multi-task collaborative training method; in the two-stage multi-task collaborative training method, the Mel-spectrogram reconstruction, pitch contour reconstruction, and adversarial speaker classification started in the first stage are used to preliminarily train the decoupling ability of the model's speech components, and the content feature reconstruction and content-rhythm feature mutual information variational upper bound minimization started in the second stage further optimize the model decoupling performance on the basis of the first stage, thereby improving the convergence efficiency of the model; the objective function used in the training method is obtained by the weighted summation of six groups of loss functions L1 to L6, and each time the mutual information variational upper bound minimization training task is performed, the mutual information parameterized network therein is subjected to five maximum likelihood trainings; The loss functions L1 and L2 are the reconstruction loss functions of Mel spectrum and pitch contour respectively, and the specific expressions are as follows: Where: X is the Mel spectrum of the original speech, P is the pitch contour of the original speech, and E() is the expectation function; The loss functions L3 and L4 are the classification loss functions of the positive classifier and the negative classifier respectively, and the specific expressions are as follows: Where: I() is an indicator function, i.e., it takes 1 when the relationship in the brackets is true and 0 when it is false; k The probability value of the positive classifier or negative classifier output prediction result belonging to the kth class, K is the number of speakers in the dataset, Numt rue The real speaker number corresponding to the input sample; The loss function L5 is the reconstruction loss function of the content feature, and its expression is as follows: Where: c rec Mel Score X rec Input to the content encoder to reconstruct the content feature encoding matrix, E() is the expectation function, c is the content feature encoding matrix reconstructed by randomly resampled Mel spectrum input to the content encoder; The loss function L6 is a loss function that minimizes the upper bound of the variational mutual information of content-rhythm features, and its expression is as follows: Where: N is the number of samples in a batch, i and j are sample index numbers, q θ (c i |r i ) is the real distribution p(c i |r i ) The mutual information variation approximate distribution obtained, the true distribution p(c i |r i ) indicates that given r i Under the conditions c i The conditional probability, r i is the rhythm feature encoding matrix of the i-th sample, c i is the content feature encoding matrix of the i-th sample; (4) Use pwCCA to evaluate the decoupling ability of each speech component of the model, adjust the weights between each decoupling loss based on the evaluation results, and retrain the model for iterative optimization; (5) According to the actual speech component conversion requirements, the pitch contours and Mel spectra of the source and target speech are input into the trained model to reconstruct the new Mel spectra X rec , and then X rec Input into a pre-trained vocoder to generate the required converted speech.

2. The multi-component speech conversion method according to claim 1, wherein: In the step (1), the pitch contour of each speech in the data set is first extracted, and then the pitch enhancement processing is performed on each speech. The mel spectra of the speech before and after the processing are respectively extracted to obtain the mel spectra of the original speech and the mel spectra of the pitch enhanced speech.

3. The multi-component speech conversion method according to claim 1, wherein: The joint random resampling module first randomly crops the input signal, and then uses a random factor to guide the deformation of each signal segment and splices them together. The value of the random factor ranges from 0.5 to 1.

5. For the same set of samples, the pitch contour and the Mel spectrum use the same random factor to ensure that the content encoding and pitch encoding are aligned in the time frame after random resampling.

4. The multi-component speech conversion method according to claim 1, wherein: The multi-channel encoder group includes a rhythm encoder, a content encoder, a timbre encoder and a pitch encoder. The output of each encoder is downsampled in the time dimension. The input of the pitch encoder is the pitch contour after random resampling, and the output is a pitch feature coding matrix f of 64×T size; the input of the content encoder is the Mel spectrum after random resampling, and the output is a content feature coding matrix c of 16×T size; the input of the rhythm encoder is the Mel spectrum with enhanced pitch, and the output is a rhythm feature coding matrix r of 2×T size; the input of the timbre encoder is the Mel spectrum of the original speech, and the output is a timbre feature coding matrix u of 256×1 size, where T is the number of frames of the speech in the time dimension.

5. The multi-component speech conversion method according to claim 4, wherein: The adversarial speaker classifier group includes two classifiers: a positive classifier and a negative classifier, wherein the input of the positive classifier is u, and the input of the negative classifier is a tensor obtained by concatenating c, r, and f. The input end of the negative classifier is additionally provided with a gradient reversal layer.

6. The multi-component speech conversion method according to claim 1, wherein: In order to evaluate the detangling capability between features, the step (4) first performs pwCCA calculations between two of the three temporal encoders. For any encoder E i The feature independence of pwCCA and the other two encoders E j and E k The average value is used to characterize, specifically: CCInd(E i )=[CCA(E i ,It j )+CCA(E i ,It k )] / 2 Among them: CCAind(E i ) is the encoder E i The feature independence evaluation index, CCA(E i ,E j ) is the encoder E i With E j The pwCCA score between i ,E k ) is the encoder E i With E k The pwCCA scores between,i,j,k∈[f,c,r], where f,c,r correspond to pitch, content, and rhythm; Based on the above evaluation results, the weights of the loss functions L2 to L6 are reset, and the model is retrained for iterative optimization.

Images