Zero sample voice conversion method based on tone decoupling
Through the combination of phased training and multi-scale tone encoder, the problem of tone leakage and tone extraction in zero-sample speech conversion is solved, and the high similarity and robustness of speech conversion is achieved.
Patent Information
- Application Number
- CN202510674116.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-23
- Publication Date
- 2025-08-26
AI Technical Summary
There are timbre leakage in the existing zero-sample voice conversion technology and inaccurate extraction of timbre features of target speakers, which affects the similarity between the output voice of the model and the target speakers.
The content decoupling strategy of phased training is adopted, combined with adversarial speaker classifiers and random data perturbation technology, and pure semantic content and accurate tone characteristics are extracted through multi-scale tone encoder and consistency loss function to generate speech with randomly changing tone but unchanged semantic content.
It effectively avoids tone leakage, improves the robustness of voice conversion and the accuracy of tone extraction, and ensures that the converted voice is highly similar to the target speaker.
Smart Images
Figure CN120544586A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of voice conversion, and in particular to a zero-sample voice conversion method based on timbre decoupling. Background Art
[0002] With the rapid development of artificial intelligence (AI), speech conversion has become a key technology in highly expressive human-computer interaction. The primary goal of speech conversion is to transform the acoustic features of a source speech (the speech to be converted) into the speech features of a target speaker, making it sound like the target speaker's pronunciation while preserving the semantic content. With the continuous advancement of technology, the research direction of speech conversion has gradually transitioned from traditional single-sample speech conversion to the more challenging zero-shot speech conversion that is more closely aligned with practical application needs. The zero-shot speech conversion task involves providing only a single speech sample of the target speaker (the reference speech). Based on this reference speech, the model can convert the source speech (the speech to be converted) into the speech of the target speaker, maintaining the semantic content consistent with the source speech, even if the speech conversion model was not fed with the target speaker's speech sample during training. This technology is of great significance in many practical application scenarios, such as film dubbing, personalized speech synthesis, speaker anonymization, and helping patients with dysarthria recover their natural speech.
[0003] With the rapid development of neural networks and deep learning methods, speech conversion technology has made significant progress, particularly methods based on convolutional neural networks, generative adversarial networks, and variational autoencoders. In recent years, speech conversion methods that use self-supervised pre-trained speech models to extract semantic content have become mainstream. These self-supervised models, with their robustness and good generalization, have enabled speech conversion technology to move away from traditional text-annotated data, gaining widespread recognition. However, these methods also face a significant challenge: the extracted semantic content still contains the timbre characteristics of the source speaker, resulting in the converted speech retaining the original speaker's voice characteristics, a phenomenon known as "timbre leakage." While previous studies have attempted to remove source speaker information from semantic content through techniques such as information bottleneck layers, vector quantization, and data perturbation, robust content decoupling strategies are lacking in complex application scenarios. Furthermore, zero-shot speech conversion techniques face the challenge of inaccurate timbre feature extraction of the target speaker, which directly affects the similarity between the model output and the target speaker. In order to improve the performance of zero-shot speech conversion, a lot of research has been carried out in recent years on content decoupling and timbre modeling. However, there are still some key issues that need to be solved. Summary of the Invention
[0004] To solve the technical problems existing in the above-mentioned prior art, the present invention proposes a zero-sample speech conversion method based on timbre decoupling, which realizes the conversion of the sound style features in the source speech (the speech to be converted) to the target speaker using only one speech of any target speaker as a reference, while keeping the semantic content consistent with the source speech.
[0005] To achieve the above object, the present invention provides a zero-sample speech conversion method based on timbre decoupling, comprising:
[0006] Obtain the speech to be converted and the reference speech of the target speaker respectively;
[0007] Build a zero-shot speech conversion model;
[0008] Inputting the speech to be converted and the reference speech of the target speaker into the zero-shot speech conversion model for processing, and outputting the conversion result;
[0009] The zero-shot speech conversion model includes a content decoupling module, a timbre decoupling and extraction module, and a speech conversion and reconstruction module. The content decoupling module and the timbre decoupling and extraction module are processed in parallel, respectively used to extract content features and timbre features and output fused features after feature fusion operation. The fused features are input to the speech conversion and reconstruction module.
[0010] The content decoupling module generates speech with randomly changing timbre but unchanged semantic content, the timbre decoupling and extraction module extracts the timbre in the target speaker's reference speech and strengthens the timbre representation, and the speech after timbre conversion is generated based on the speech conversion and reconstruction module.
[0011] Preferably, generating the speech with randomly changing timbre but unchanged semantic content comprises:
[0012] Input the speech to be converted into a pre-trained speech model WavLM to extract a self-supervised representation, and generate a semantic content representation through a content encoder;
[0013] An adversarial speaker classifier is introduced, and a gradient reversal layer is connected before the adversarial speaker classifier to remove the source speaker identity information;
[0014] Applying random data perturbation to the speech with the source speaker's identity information removed to generate speech with the randomly changed timbre but the unchanged semantic content;
[0015] The random data perturbation includes at least one of random pitch shift, random formant shift and spectrum reshaping, and a data replay strategy is used to alleviate catastrophic forgetting.
[0016] Preferably, extracting the timbre of the target speaker's reference speech and enhancing the timbre representation by the timbre decoupling and extraction module includes:
[0017] Performing Fourier transformation on the reference speech of the target speaker to obtain an original Mel-spectrogram, inputting the original Mel-spectrogram into a multi-scale timbre encoder, and extracting a first timbre embedding feature;
[0018] Randomly cropping and reorganizing the original mel-spectrogram to generate a data-enhanced second mel-spectrogram with changed content but consistent timbre, and inputting the second mel-spectrogram into the multi-scale timbre encoder to extract a second timbre embedding feature;
[0019] A multi-level consistency loss function is applied to constrain the first timbre embedding feature and the second timbre embedding feature to obtain accurate timbre features.
[0020] Preferably, the multi-scale timbre encoder comprises:
[0021] One-dimensional convolution groups with multi-sized convolution kernels are used to extract fine-grained timbre features at different scales;
[0022] Gated convolution module, used to model temporal dependencies by gradually increasing the convolution kernel size;
[0023] Statistical attention pooling layer to enhance global timbre representation;
[0024] Among them, the one-dimensional convolution group of multi-size convolution kernels, the gated convolution module and the statistical attention pooling layer are connected in sequence.
[0025] Preferably, the multi-level consistency loss function consists of a single sample level consistency loss and a training batch class center level consistency loss.
[0026] Preferably, generating the voice after timbre conversion based on the voice conversion and reconstruction module includes:
[0027] The Mel-spectrogram of the initial reference speech is input into a multi-scale timbre encoder to extract the target timbre embedding;
[0028] The semantic content representation of the speech to be converted is integrated with the target timbre embedding, a posterior distribution is generated through a reversible standard flow model, and the posterior distribution and the target timbre embedding are input into an inverse Fourier transform vocoder to reconstruct the converted speech.
[0029] Preferably, the speech to be converted and the reference speech of the target speaker are input together into the zero-sample speech conversion model for processing, and the process also includes: performing a speech conversion inference test and processing the converted speech through a t-SNE visualization network.
[0030] Compared with the prior art, the present invention has the following advantages and technical effects:
[0031] 1. The present invention adopts a content decoupling strategy of phased training. Through a continuous two-stage training strategy, the model integrates the advantages of two content decoupling technologies, namely the adversarial speaker classifier and the random data perturbation, to achieve more robust extraction of pure semantic content and effectively avoid the occurrence of the "timbre leakage" phenomenon. In addition, the present invention extracts a self-supervised representation based on the pre-trained speech model WavLM. The representation contains rich semantic information, which ensures the accuracy of the model in content extraction and its generalization in complex application scenarios. The present invention specifically selects the middle layer output of the WavLM hidden layer as the source of content representation. After investigation and experiments, it has been proved that the middle layer contains richer and relatively pure semantic content information.
[0032] 2. The present invention applies consistency loss functions at the single sample level and the speaker-centric level of the training batch respectively, and adopts a consistency learning architecture for training. It guides the timbre encoder to decouple timbre and content representation from two Mel-spectra with consistent timbre but different content, focusing on the modeling of timbre information while ignoring the interference of content information, effectively improving the accuracy of timbre extraction.
[0033] 3. This paper proposes a multi-scale timbre encoder specifically for timbre modeling, which offers significant advantages over the speaker verification model used as a timbre extractor in other approaches. The encoder includes a convolution group composed of multiple kernel sizes, which extracts fine-grained features across different receptive fields and dynamically fuses them. Subsequently, the time series is modeled using three consecutive gated convolution group modules with different convolution sizes. The gating mechanism optimizes information flow, allowing the model to focus more on timbre extraction and preventing content from interfering with timbre modeling. A statistical pooling module further enhances timbre representation. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] The accompanying drawings, which constitute part of this application, are intended to provide a further understanding of this application. The exemplary embodiments and descriptions of this application are intended to explain this application and do not constitute an improper limitation on this application. In the accompanying drawings:
[0035] Figure 1 Schematic diagram of a model of zero-sample speech conversion based on timbre decoupling according to an embodiment of the present invention;
[0036] Figure 2 Schematic diagram of a timbre decoupling and extraction method with multi-level consistency constraints according to an embodiment of the present invention;
[0037] Figure 3 Schematic diagram of a multi-scale convolution group module belonging to a multi-scale timbre encoder according to an embodiment of the present invention;
[0038] Figure 4 Schematic diagram of a gated convolution module of a multi-scale timbre encoder according to an embodiment of the present invention;
[0039] Figure 5 Schematic diagram of the t-SNE visualization results of the converted speech according to an embodiment of the present invention, wherein (a) is a visualization result diagram of the converted speech according to an embodiment of the method for removing timbre decoupling according to an embodiment of the present invention, and (b) is a visualization result diagram of the converted speech according to an embodiment of the present invention. DETAILED DESCRIPTION
[0040] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.
[0041] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0042] This embodiment proposes a zero-sample speech conversion method based on timbre decoupling, including:
[0043] Get the voice to be converted;
[0044] Obtain the reference speech of the target speaker;
[0045] Build a zero-shot speech conversion model;
[0046] Inputting the speech to be converted and the reference speech of the target speaker into the zero-shot speech conversion model for processing, and outputting the conversion result;
[0047] The zero-shot speech conversion model includes a content decoupling module, a timbre decoupling and extraction module, and a speech conversion and reconstruction module. The content decoupling module and the timbre decoupling and extraction module are processed in parallel, respectively used to extract content features and timbre features and output fused features after feature fusion operation. The fused features are input to the speech conversion and reconstruction module.
[0048] The content decoupling module generates speech with randomly changing timbre but unchanged semantic content, the timbre decoupling and extraction module extracts the timbre in the target speaker's reference speech and strengthens the timbre representation, and the speech after timbre conversion is generated based on the speech conversion and reconstruction module.
[0049] Furthermore, the speech to be converted and the reference speech of the target speaker are input into the zero-shot speech conversion model for processing, including:
[0050] A speech with randomly changing timbre but unchanged semantic content is generated through the content decoupling module, the timbre in the target speaker's reference speech is extracted and the timbre representation is enhanced through the timbre decoupling and extraction module, and the timbre-converted speech is generated based on the speech conversion and reconstruction module.
[0051] Furthermore, we generate speech with random timbre changes but unchanged semantic content, including:
[0052] The speech to be converted is input into the pre-trained speech model WavLM to extract the self-supervised representation, and the semantic content representation is generated through the content encoder;
[0053] An adversarial speaker classifier is introduced in the first stage of model training. The source speaker identity information is removed by connecting a gradient reversal layer before the speaker classifier.
[0054] In the second stage of model training, random data perturbations are applied to the source speech to generate speech with random variations in timbre but unchanged semantic content.
[0055] The random data perturbation includes at least one of random pitch shift, random formant shift and spectrum reshaping, and a data replay strategy is used to alleviate catastrophic forgetting.
[0056] Specifically, this embodiment proposes a content decoupling strategy with phased training. Through the phased training strategy, combined with the technical advantages of adversarial speaker classifier and random data perturbation in content decoupling, the robustness of the network in extracting pure semantic content information is enhanced, and "timbre leakage" is effectively avoided in complex real-world scenarios.
[0057] In this embodiment, an adversarial speaker classifier is introduced, whose goal is to accurately classify speakers based on speaker characteristics carried in the input. By inserting a gradient reversal layer before the speaker classifier, the content encoder receives gradient signals in the opposite direction during training. This forces the content encoder to remove speaker identity information from its output, preventing it from generating discriminative representations for speaker classification.
[0058] Figure 1 The overall framework of the zero-shot speech conversion network based on this strategy is presented, with the model training process on the left and the inference and testing process on the right. During the training phase, network parameters are jointly optimized using a multi-level consistency loss function, a reconstruction loss function, a divergence loss function, a discriminator loss function, and a feature matching loss function. During the testing phase, the optimal trained model is used with fixed network parameters. The source speech and the target speaker's reference speech are input into the model. The generated converted speech is consistent with the target speech in timbre and style, while the content information is consistent with the source speech.
[0059] The processing of the content decoupling module specifically includes:
[0060] In the first stage, the source speech (to be converted) is first extracted from a self-supervised representation using the pre-trained speech model WavLM. This is then fed into a content encoder to extract a semantic content representation. Because this extracted semantic content representation still contains information about the source speaker's voice, it is then fed into an adversarial speaker classifier for classification. Due to the adversarial training objective of the gradient reversal layer, the content encoder is forced to gradually discard information related to the source speaker's identity, thereby extracting a pure semantic content representation.
[0061] In the second phase, the model inherits the learnable parameters from the first phase, but no longer uses the adversarial speaker classifier module. The source speech (the speech to be converted) is processed using random data perturbation technology to generate speech with randomly varying timbre but unchanged semantic content.
[0062] This embodiment employs three random data perturbation techniques: random pitch shifting, random formant shifting, and spectrum reshaping. These techniques directly remove timbre information from the source speech, increasing the complexity of the data. Training the model on these enhanced complex samples effectively improves the model's generalization capabilities, significantly contributing to performance gains. Furthermore, this embodiment incorporates a data replay strategy to mitigate the catastrophic forgetting problem faced by staged training models.
[0063] Furthermore, the timbre decoupling and extraction module extracts the timbre of the target speaker's reference speech and enhances the timbre representation, including:
[0064] The target speaker's reference speech is subjected to Fourier transformation to obtain an original Mel-spectrogram, and the original Mel-spectrogram is input into a multi-scale timbre encoder to extract a first timbre embedding feature;
[0065] Randomly cropping and reorganizing the original mel-spectrogram to generate a data-enhanced second mel-spectrogram with changed content but consistent timbre, and inputting the second mel-spectrogram into the multi-scale timbre encoder to extract a second timbre embedding feature;
[0066] Applying a multi-level consistency loss function to constrain the first timbre embedding feature and the second timbre embedding feature can guide the multi-scale timbre encoder to focus on extracting timbre and avoid the interference of content on the process, thereby obtaining accurate timbre features.
[0067] Specifically, the content information and timbre representation in speech signals are often entangled, interfering with timbre modeling and resulting in inaccurate timbre content extraction. To address this issue, this embodiment proposes a timbre decoupling and extraction method based on multi-level consistency constraints. This method decouples content and timbre information, and uses a multi-level consistency constraint loss function to guide the timbre encoder to focus on timbre extraction while avoiding interference from content information, thereby improving the accuracy of timbre extraction.
[0068] The original Mel spectrogram obtained by Fourier transform of the source speech is first input into the multi-scale timbre encoder to extract the timbre embedding feature g a Subsequently, the original Mel spectrogram is randomly cropped and reorganized to generate a data-enhanced Mel spectrogram with changed content but consistent timbre, which is then input into a parameter-shared multi-scale timbre encoder to extract the timbre embedding g b Since the timbre features contained in the original and data-enhanced Mel spectrograms are globally consistent, a multi-level consistency loss function is applied to constrain the timbre embedding g a and g b The difference between the two, thus guiding the timbre encoder to focus on timbre extraction and ignore the interference of content information. The network updates the parameters according to the calculated loss value and gradually optimizes the model.
[0069] The multi-level consistency constraint consists of a single-sample level consistency loss and a training batch class center level consistency loss, which is applied to the training of the multi-scale timbre encoder to enable it to focus on timbre modeling.
[0070] The main architecture of this embodiment is based on the end-to-end speech synthesis model VITS (Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech), which is a variational autoencoder architecture. At the same time, the quality of speech reconstruction is enhanced by adversarial generative training methods. This architecture can provide speech synthesis quality with the naturalness of real speech.
[0071] The use of a vocoder based on inverse short-time Fourier transform effectively reduces the amount of computation compared to the mainstream model using the HiFiGAN vocoder, and improves the real-time inference speed of the model deployed on the terminal.
[0072] Furthermore, the multi-scale timbre encoder includes:
[0073] One-dimensional convolution groups with multi-sized convolution kernels are used to extract fine-grained timbre features at different scales;
[0074] Gated convolution module, used to model temporal dependencies by gradually increasing the convolution kernel size;
[0075] Statistical attention pooling layer to enhance global timbre representation;
[0076] Among them, the one-dimensional convolution group of multi-size convolution kernels, the gated convolution module and the statistical attention pooling layer are connected in sequence.
[0077] Specifically, the multi-scale timbre encoder consists of a one-dimensional convolutional group with kernels of varying sizes, which extracts fine-grained features at different scales. This is followed by a gated convolutional module with progressively larger kernel sizes to model the time series, and finally, a statistical attention pooling layer enhances the timbre representation.
[0078] Furthermore, generating voice after timbre conversion based on the voice conversion and reconstruction module includes:
[0079] The Mel-spectrogram of the initial reference speech is input into a multi-scale timbre encoder to extract the target timbre embedding;
[0080] The semantic content representation of the speech to be converted is integrated with the target timbre embedding, and a posterior distribution is generated through a reversible normalizing flow model. The posterior distribution and the target timbre embedding are input into an inverse Fourier transform vocoder to reconstruct the converted speech. The normalizing flow model is a generative model that can reversibly convert a simple distribution (such as a normal distribution) into a complex distribution.
[0081] Furthermore, the speech to be converted and the target speaker's reference speech are input into the zero-shot speech conversion model for processing, which also includes: performing a speech conversion inference test and processing the converted speech through a t-SNE visualization network.
[0082] Specifically, during the inference and testing phase, the target speaker's reference signal is first converted into a mel-spectrogram and fed into a multi-scale timbre encoder to extract a timbre embedding. Subsequently, the source speech is fed into a pre-trained speech model, WavLM, to obtain a self-supervised representation, which is then fed into a content encoder to extract semantic content information.
[0083] The semantic content information is feature fused with the target timbre embedding and input into the inverse standard flow model, where multiple coupling layers are used to transform the distribution and generate the posterior distribution.
[0084] The posterior distribution and timbre embedding are input into the inverse Fourier transform vocoder to reconstruct the converted speech.
[0085] Taking a batch of converted samples as an example, a t-SNE visualization is plotted. Converted speech from the same target speaker, due to their similar timbre embeddings, clusters together in the feature embedding space. This result demonstrates the effectiveness of this embodiment in improving speaker similarity and enhancing the robustness of speech conversion.
[0086] Finally, to verify the effectiveness of the proposed method in this example, we removed this method and retrained the model. We then conducted inference testing using the same method and plotted the converted speech as a t-SNE visualization. Comparing the two sets of t-SNE plots reveals that this example significantly improves both timbre modeling and the similarity of voice conversion.
[0087] The goal of the zero-shot speech conversion task is to convert features such as timbre in the source speech (the speech to be converted) into the target speech using only one speech of the target speaker as a reference, while keeping the semantic content of the converted speech consistent with that of the source speech.
[0088] To make the purpose, technical solutions, and advantages of this embodiment more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. The described embodiments are part of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.
[0089] Example 1
[0090] This example proposes a zero-shot speech conversion method based on timbre decoupling. Its implementation process mainly includes three key modules: content decoupling module, timbre decoupling and extraction module, and speech conversion and reconstruction module.
[0091] Figure 1 The overall framework of the zero-shot speech conversion network based on this strategy is presented, with the model training process on the left and the inference and testing process on the right. During the training phase, network parameters are jointly optimized using a multi-level consistency loss function, a reconstruction loss function, a divergence loss function, a discriminator loss function, and a feature matching loss function. During the testing phase, the optimal trained model is used, the network parameters are fixed, and the source speech and the target speaker's reference speech are input into the model. The generated converted speech not only matches the target speech in timbre and style, but also maintains the same content information as the source speech.
[0092] Part 1: Implementation of the content decoupling strategy for phased training:
[0093] In past research on speech conversion, the use of gradient inversion-based speaker classifiers and random data perturbations has been proven to be effective in most scenarios for decoupling speaker-irrelevant content information from speech. However, in complex application scenarios, using only one content decoupling technique to extract semantic content makes speech conversion less robust and prone to "timbre leakage", resulting in a low similarity between the converted speech and the target speaker. Combining the two content decoupling techniques mentioned above can achieve a more robust content decoupling effect, but there are obstacles to directly combining the two techniques because random data perturbations will destroy the speaker identity features required by the speaker classifier. Therefore, this embodiment proposes a phased training strategy, which aims to combine the advantages of the above two techniques to achieve pure content extraction.
[0094] The specific implementation plan is as follows:
[0095] like Figure 1 As shown in Figure 1, in the first stage of training (Stage 1), an adversarial speaker classifier is used to guide the content encoder to remove source speaker-related information. Specifically, the source speech y is input into the pre-trained speech model WavLM to extract a self-supervised representation, which is then input into the content encoder to extract a content representation. However, the content representation inevitably contains source speaker-related information. Therefore, the extracted content representation is further input into the adversarial speaker classifier module. The adversarial training objective forces the content encoder to remove speaker-related information during training, thereby focusing on content extraction.
[0096] In the second stage of training (Stage 2), the model inherits the learnable parameters from the first stage, but the adversarial speaker classification branch is no longer used. The source speech y is first perturbed with random data to produce speech y' with randomly altered timbre but unchanged semantic content. To increase the diversity of timbre distortion styles, three random data perturbation techniques are used: random pitch shifting, random formant shifting, and random shaped mel-spectrograms.
[0097] Specifically, for each source speech y, one of the random data perturbation techniques is randomly selected to produce an enhanced speech with a random timbre but unchanged content. Furthermore, to mitigate the catastrophic forgetting problem commonly faced by models trained on decomposed segments, a data replay strategy is employed. Specifically, the parameters of the three applied data perturbation techniques are configured so that, when implemented, they all have a certain probability of outputting the source speech y without changing the timbre and being reused in the second stage. This facilitates the model's ability to decouple content from the speaker classifier trained against it in the first stage.
[0098] Part II: Implementation of the multi-level consistency constrained timbre decoupling and extraction method:
[0099] Accurate extraction of timbre is crucial for speech conversion tasks. The content in the speech signal is often mixed and entangled with the timbre representation, which interferes with timbre modeling. To solve this problem, this embodiment proposes a multi-level consistent timbre decoupling and extraction strategy to detangle timbre from content, allowing the timbre encoder to focus on timbre modeling. Figure 2 , Figure 2 The proposed multi-level consistent timbre decoupling strategy is presented. This strategy adopts a consistent learning architecture. The original Mel-spectrogram and the Mel-spectrogram reorganized after random cropping pass through a parameter-sharing timbre encoder, and output timbre embeddings g and g' respectively. Since their speaker timbre representations are consistent but the speech content is different, applying consistency constraints to the Mel-spectrogram at the single sample level and the training batch class center level can enable the network to guide the timbre encoder to disentangle timbre from content during training and focus on speaker timbre extraction. In addition, in order to achieve more robust timbre modeling and extraction, a multi-scale timbre encoder is proposed. This multi-level consistent timbre decoupling and extraction strategy is mainly jointly trained under the constraints of the above-mentioned multi-level consistency loss function. The training speed is related to the configuration of the machine hardware and the size of the sample.
[0100] Step 3: Test the model’s speech conversion performance and visualize the model’s test results:
[0101] Step 1: Input the source speech and target speech into the inference test model respectively;
[0102] Specifically, the source speech is input into the self-supervised pre-trained speech model WavLM to extract self-supervised representation, and the target speaker's speech is converted into a Mel-spectrogram and input into a multi-scale timbre encoder to extract timbre features.
[0103] Step 2: Extract content representation and fuse the target timbre;
[0104] The self-supervised representation is further input into the content encoder to extract content information, and then input into the reversible standard flow model together with the extracted timbre features of the target speaker. The multi-layer coupling layer fuses the content information and the timbre features of the target speaker and converts them into a posterior distribution that can be used by the vocoder to reconstruct the speech.
[0105] Step 3: Reconstruct the converted speech;
[0106] Specifically, the posterior distribution containing the source speech content information and the target speaker's timbre characteristics is input into an inverse Fourier transform vocoder to generate converted speech. This speech is consistent with the source speech in terms of content information, but sounds like the pronunciation of the target speaker.
[0107] Step 4: Select a batch of test samples from multiple speakers for voice conversion inference testing and process the converted speech through a t-SNE visualization network. First, use a pre-trained speaker verification model to extract speaker embeddings for the converted speech. Then, use a t-SNE network to map these embeddings into a low-dimensional space. Conversion samples from the same target speaker show clustering in the embedding space, demonstrating the robustness of this embodiment's voice conversion model.
[0108] Example 2
[0109] Specific description of zero-sample speech conversion based on timbre decoupling:
[0110] 1. Model architecture selection:
[0111] The model architecture of this embodiment is based on the VITS speech synthesis model. This architecture is chosen for its excellent speech reconstruction capabilities, which are fundamental requirements for speech conversion tasks, prioritizing naturalness and intelligibility of the converted speech. This architecture uses a variational autoencoder combined with a reversible standard flow model to achieve the conversion between prior and posterior distributions.
[0112] Specifically, the content information output by the content extractor is treated as a priori distribution after the distribution generated by the mapping layer, and the linear spectrum x lin The distribution output by the posterior encoder is considered as the posterior distribution. During the training phase, the posterior distribution is converted into a priori distribution through the multi-layer coupling layer of the standard flow model, and the KL divergence loss L is used. kl To align the prior distribution and the posterior distribution coupled with the standard flow model. At the same time, the latent variable z is sampled from the posterior distribution and input into the inverse short-time Fourier transform vocoder to reconstruct the source speech, and the reconstruction loss function L is used. rec The error between the source speech and the reconstructed speech is calculated. In addition, to further improve the quality of the reconstructed speech, a discriminator is introduced to enhance the network's reconstruction ability through a generative adversarial training model, thereby improving the naturalness of the generated speech.
[0113] 2. Methods:
[0114] This embodiment proposes a multi-level consistency-constrained timbre decoupling and extraction method for zero-shot speech conversion, aiming to improve speaker similarity in speech conversion. This method primarily involves a phased training content decoupling strategy to improve the robustness of semantic content information extraction. The multi-level consistency-constrained timbre decoupling and extraction method avoids content interference with timbre modeling, thereby improving timbre modeling accuracy. Furthermore, a multi-scale timbre encoder specifically for timbre modeling is proposed, which comprehensively and accurately extracts timbre features at both local and global scales. Its effectiveness and generalizability are experimentally verified. To validate speech conversion performance, in addition to using subjective mean opinion scores (MOS) to measure the naturalness and similarity of the converted speech, a t-SNE visualization network is used to map the converted speech samples into an embedding space, visually demonstrating that this embodiment improves speaker similarity in speech conversion. Furthermore, an ablation control group is used to verify the effectiveness of the proposed multi-level consistency timbre decoupling and extraction strategy.
[0115] The following focuses on describing several key modules in the model involved:
[0116] 1. Content decoupling strategy for phased training (content decoupling module):
[0117] Random data perturbation technology changes the timbre information of the source speech and also changes the speaker identity characteristics of the source speech. Therefore, the combined use of adversarial speaker classifiers and random data perturbation technology faces challenges. In order to achieve more robust semantic content extraction, this embodiment proposes a phased content training strategy, which effectively combines the above two technical advantages of content decoupling. In addition, the model applies a data replay strategy in the second stage of training to avoid the catastrophic forgetting problem that is prone to be encountered in staged training models. The specific implementation details are as follows:
[0118] like Figure 1 As shown in the figure, in the first training phase, an adversarial speaker classifier is used to guide the content encoder to remove information related to the source speaker. Specifically, the source speech y is extracted from a supervised representation using the pre-trained speech model WavLM, which is then fed into the content encoder to generate a semantic content representation. Since the extracted semantic content representation is inevitably contaminated with information related to the source speaker's voice, the adversarial training objective forces the content encoder to gradually remove information related to the source speaker's identity during training, thereby extracting pure semantic content information.
[0119] In the second phase of training, the model inherits the learnable parameters from the first phase, but the adversarial speaker classifier branch is no longer used. The source speech y is first perturbed with random data to generate y', which has altered timbre characteristics but unchanged semantic content. Three random data perturbation techniques are used: random pitch shifting, random formant shifting, and random reshaping of the mel-spectrogram. During implementation, one random data perturbation technique is randomly applied to the source speech, altering its timbre and directly removing the speaker style characteristics of the source speech.
[0120] 2. Timbre decoupling and extraction method with multi-level consistency constraints (timbre decoupling and extraction module):
[0121] Accurate extraction of timbre features is crucial for speech conversion tasks. However, the content information in speech signals is often mixed with timbre representation, so the content interferes with the modeling of timbre features. To this end, a timbre decoupling and extraction strategy based on multi-level consistency constraints is proposed, such as Figure 2 As shown in Figure 2, the strategy is based on the architecture of consistency learning. In the upper branch, the Mel spectrum x mel Input to the multi-scale timbre encoder to extract the timbre embedding g. In the next branch, x mel After scrambling and reorganizing (SR), we get the Mel spectrogram x′ with the timbre preserved but the content scrambled mel , which is then fed into the multi-scale timbre encoder to output the timbre embedding g′. Since the timbre features represented by g and g′ originate from the same speech, using multi-level consistency constraints between g and g′ can guide the multi-scale timbre encoder module to focus on timbre feature extraction.
[0122] Random cropping and reassembly operations:
[0123] Speech signals can be considered to be composed of static (such as timbre) and dynamic (such as content) components on a temporal scale. In addition, experiments have shown that the timbre characteristics of audio clips obtained by randomly cropping speech signals on the time axis are completely preserved. Based on the above observations, this embodiment crops and randomly reorganizes the Mel spectrum of the source speech to generate a Mel spectrum x′ with shuffled content but unchanged timbre. mel .
[0124] Loss function for multi-level consistency constraints:
[0125] Even different voices belonging to the same speaker often output inconsistent timbre embeddings after passing through a timbre encoder. This is because the timbre extraction process is affected by factors such as content and noise, resulting in unstable speaker similarity after voice conversion. To alleviate this problem, a timbre decoupling strategy was proposed. This strategy applies a consistency loss at both the single-sample level and the training batch class center level. This aims to guide the multi-scale timbre encoder to decouple timbre from the target voice, minimizing the interference of content information on timbre modeling.
[0126] At the single-sample level, the mean squared error (MSE) loss function is used to minimize the error between g and g′.
[0127] At the training batch class center level, first obtain the class center of the timbre feature embedding of each speaker in the current training batch. Similarly, calculate the corresponding timbre embedding class center μ′ of the lower branch c Then, applying the center loss function to the batch center of the upper and lower branch timbre representation can make the timbre embeddings of different samples of the same speaker more closely distributed in the embedding space.
[0128] Multi-scale Timbre Encoder:
[0129] like Figure 2 As shown in , the multi-scale timbre encoder aims to achieve efficient timbre extraction. Its main structure consists of a multi-scale convolution group (such as Figure 3 As shown in ), multiple one-dimensional convolutions with different convolution kernel sizes are used to extract fine-grained timbre features in different receptive fields. Subsequently, to model temporal dependencies, three gated convolution modules with increasing convolution and size are used (as shown in Figure 4 (as shown in Figure 2). The gating mechanism in the gated convolution, under the aforementioned multi-level consistency constraints, effectively constrains information flow, enabling the timbre encoder to decouple timbre features from content information. Finally, statistical pooling is used to further enhance the timbre representation and generate the timbre embedding g.
[0130] 3. Visualize the effect of the speech conversion algorithm (speech conversion and reconstruction module):
[0131] Speech conversion algorithms typically use subjective evaluation methods, including the naturalness mean opinion score (nMOS) and speaker similarity mean opinion score (sMOS), which measure speech naturalness and intelligibility. To more intuitively demonstrate the advantages of the proposed speech conversion algorithm, the converted test samples were mapped into an embedding space using a t-SNE network. Furthermore, to verify the effectiveness of the proposed multi-level consistency-based timbre decoupling and extraction module, the model was retrained after removing this strategy, and its output was visualized using a t-SNE network with the same configuration as a control group for comparison. The specific implementation details are as follows:
[0132] A set of test samples is set up and the proposed voice conversion algorithm is used for inference testing. The converted speech is first passed through a pre-trained speaker verification model to extract the speaker identity embedding, and then these embedding features are input into the t-SNE network and mapped into the embedding space. Among them, samples from the same target speaker are represented by the same color in the embedding space. Figure 5 (a)- Figure 5As shown in (b), speech samples of the same target speaker are well clustered together in the embedding space, indicating that the proposed voice conversion model has robust voice conversion performance.
[0133] In order to verify the effectiveness of the proposed multi-level consistent timbre decoupling and extraction method, a control experiment was set up to remove the strategy and retrain the model. Under the same test conditions, the visualization results were obtained by configuring a consistent t-SNE network. Figure 5 (a)- Figure 5 (b), The comparison results show that the adoption of this strategy significantly improves the robustness of speech conversion.
[0134] In summary, the advantages of this embodiment are: the converted speech has a high degree of similarity, and the conversion process exhibits good robustness. The proposed phased training content decoupling strategy can effectively extract semantic content information unrelated to the source speaker, and the multi-level consistent timbre decoupling and extraction method significantly improves the accuracy of timbre modeling and extraction. In addition, this embodiment has speech synthesis capabilities close to real speech and has advantages in real-time inference speed.
[0135] The zero-sample speech conversion method based on timbre decoupling proposed in this embodiment effectively decouples timbre from content information, and has accurate and generalizable timbre extraction capabilities. Furthermore, this embodiment has a more robust content decoupling capability, effectively avoiding the "timbre leakage" problem. The use of an inverse short-time Fourier transform vocoder significantly reduces the model's computational complexity and improves the model's inference speed. Combining these advantages, this embodiment can achieve speech conversion with a high degree of speaker similarity, and the naturalness of the synthesized speech is close to that of real speech.
[0136] The above are merely preferred embodiments of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.
Claims
1. A zero-sample speech conversion method based on timbre decoupling, characterized in that: include: Obtain the speech to be converted and the reference speech of the target speaker respectively; Build a zero-shot speech conversion model; Inputting the speech to be converted and the reference speech of the target speaker into the zero-shot speech conversion model for processing, and outputting the conversion result; The zero-shot speech conversion model includes a content decoupling module, a timbre decoupling and extraction module, and a speech conversion and reconstruction module. The content decoupling module and the timbre decoupling and extraction module are processed in parallel, respectively used to extract content features and timbre features and output fused features after feature fusion operation. The fused features are input to the speech conversion and reconstruction module. The content decoupling module generates speech with randomly changing timbre but unchanged semantic content, the timbre decoupling and extraction module extracts the timbre in the target speaker's reference speech and strengthens the timbre representation, and the speech after timbre conversion is generated based on the speech conversion and reconstruction module.
2. The zero-sample speech conversion method based on timbre decoupling according to claim 1, characterized in that: Generating speech with randomly changing timbre but unchanged semantic content, comprising: Input the speech to be converted into a pre-trained speech model WavLM to extract a self-supervised representation, and generate a semantic content representation through a content encoder; An adversarial speaker classifier is introduced, and a gradient reversal layer is connected before the adversarial speaker classifier to remove the source speaker identity information; Applying random data perturbation to the speech with the source speaker's identity information removed to generate speech with the randomly changed timbre but the unchanged semantic content; The random data perturbation includes at least one of random pitch shift, random formant shift and spectrum reshaping, and a data replay strategy is used to alleviate catastrophic forgetting.
3. The zero-sample speech conversion method based on timbre decoupling according to claim 1, characterized in that: The timbre decoupling and extraction module extracts the timbre of the target speaker's reference speech and enhances the timbre representation, including: Performing Fourier transformation on the reference speech of the target speaker to obtain an original Mel-spectrogram, inputting the original Mel-spectrogram into a multi-scale timbre encoder, and extracting a first timbre embedding feature; Randomly cropping and reorganizing the original mel-spectrogram to generate a data-enhanced second mel-spectrogram with changed content but consistent timbre, and inputting the second mel-spectrogram into the multi-scale timbre encoder to extract a second timbre embedding feature; A multi-level consistency loss function is applied to constrain the first timbre embedding feature and the second timbre embedding feature to obtain accurate timbre features.
4. The zero-sample speech conversion method based on timbre decoupling according to claim 3, characterized in that: The multi-scale timbre encoder comprises: One-dimensional convolution groups with multi-sized convolution kernels are used to extract fine-grained timbre features at different scales; Gated convolution module, used to model temporal dependencies by gradually increasing the convolution kernel size; Statistical attention pooling layer to enhance global timbre representation; Among them, the one-dimensional convolution group of multi-size convolution kernels, the gated convolution module and the statistical attention pooling layer are connected in sequence.
5. The zero-sample speech conversion method based on timbre decoupling according to claim 3, characterized in that: The multi-level consistency loss function consists of a single-sample level consistency loss and a training batch class center level consistency loss.
6. The zero-sample speech conversion method based on timbre decoupling according to claim 1, characterized in that: Generating voice after timbre conversion based on the voice conversion and reconstruction module includes: The Mel-spectrogram of the initial reference speech is input into a multi-scale timbre encoder to extract the target timbre embedding; The semantic content representation of the speech to be converted is integrated with the target timbre embedding, a posterior distribution is generated through a reversible standard flow model, and the posterior distribution and the target timbre embedding are input into an inverse Fourier transform vocoder to reconstruct the converted speech.
7. The zero-sample speech conversion method based on timbre decoupling according to claim 1, characterized in that: The speech to be converted and the reference speech of the target speaker are input into the zero-sample speech conversion model for processing, and the method further includes: performing a speech conversion inference test and processing the converted speech through a t-SNE visualization network.