A zero-sample-based song timbre rapid conversion method and device

Abstract

This invention discloses a method and apparatus for rapid conversion of singing voice timbre based on zero-shot sampling. The method includes constructing a singing dataset containing dry vocals and lyrics; extracting the audio codebook index sequence of the dry vocals using a singing feature decoupler incorporating a Hubert model and residual quantization codebook; introducing a text encoder to extract morpheme features and morpheme index sequences from the lyrics; optimizing the singing feature decoupler through cross-prediction to improve the accuracy of speech content feature extraction; introducing pitch and timbre features to represent prosody; and enhancing the quality of synthesized vocals generated by the generator based on speech content features, pitch features, and timbre features through adversarial training. This achieves rapid conversion of song vocal timbre to the user's timbre while ensuring conversion quality.

Description

Technical Field

[0001] This invention belongs to the field of natural language processing technology in artificial intelligence, and specifically relates to a method and apparatus for rapid conversion of singing voice timbre based on zero samples. Background Technology

[0002] Vocal timbre conversion aims to transform the timbre of a given human voice in a song into that of another person while maintaining the consistency of the song's content. Existing vocal timbre conversion methods employ few-shot learning, requiring tens of minutes to several hours of data to fine-tune the model and achieve timbre conversion. Zero-shot vocal timbre conversion, on the other hand, requires converting the timbre of the original song's human voice directly from the original song and a few seconds of reference audio without training a model. Previous research on timbre conversion has primarily focused on non-singing speech conversion, mainly categorized into synthesis methods and conversion methods.

[0003] Synthesis methods generally employ encoder-decoder deconstruction, incorporating feature decoupling techniques. Multiple encoders are used to extract content and speaker representations from speech. During training, the synthesis method first trains the speaker encoder and content encoder separately, then fine-tunes them together with the decoder. Existing research primarily utilizes Generative Adversarial Networks (GANs) training strategies, introducing a speaker determiner during encoder training to identify whether the content encoder represents the same speaker or a different speaker. By preventing the determiner from identifying the speaker, the content encoder is guaranteed to be free of speaker information, thus achieving feature decoupling. Finally, by concatenating the original content and speaker features, the decoder generates the timbre-converted speech.

[0004] The conversion method skips the feature decoupling and regeneration process, directly converting the original speech content into the target timbre. CycleGAN uses a converter to input the original speaker's speech, outputs the target speaker's speech, and then another converter converts it back into the original speaker's speech. During this process, a discriminator determines whether it is the same speaker, thus enabling adversarial training. However, CycleGAN is limited by the number of speakers, requiring a separate model to be trained for each speaker. StarGAN introduces speaker feature representations during the direct conversion process and uses a discriminator to determine whether the speaker belongs to the input speaker, thereby achieving direct timbre conversion for multiple speakers. In summary, while the conversion method avoids the feature decoupling process of the synthesis method, it still relies on speaker features to achieve specific timbre conversion. Furthermore, since speaker features are not removed from the speech content, relying solely on the discriminator for speaker identification can easily lead to mode collapse, especially in zero-shot scenarios, where it is difficult to ensure that the original speech features do not interfere with the speaker features.

[0005] Current voice conversion methods require training on large-scale speech datasets. These datasets can be categorized into two types: 1) paired datasets and 2) unparallel datasets. Paired datasets contain multiple speakers reciting the same passage. However, under these stringent requirements, existing paired datasets are scarce, making it difficult to support speech model training. Furthermore, paired data and a limited number of speakers pose a risk of overfitting, further reducing the model's generalization performance. Unparallel datasets, on the other hand, are massive, containing different speakers and different speech content. While providing ample data for speech conversion models, issues such as speaker uncertainty, multilingual speech, and lack of annotations pose significant challenges to speech alignment and speaker annotation in voice conversion models. Although there are existing speech conversion datasets such as Voice Conversion Challenge and LibriSpeech, there is a lack of large-scale datasets specifically for singing voice timbre conversion.

[0006] In summary, current data on vocal timbre conversion for singing is extremely scarce, and methods still need improvement. Direct timbre conversion methods, due to unclear feature decoupling and reliance on discriminators, are difficult to transfer to zero-shot scenarios. Synthesis methods, through a two-stage process of feature decoupling and synthesis, alleviate the feature coupling problem to some extent, but their content extraction methods are not suitable for singing scenarios. This is because singing, compared to speaking, downplays speech content and emphasizes rhythm, and the vocal style of songs leads to more diverse speech content; therefore, both the speaker encoder and the speech content encoder need adjustment. Summary of the Invention

[0007] In view of the above, the purpose of this invention is to provide a method and apparatus for fast singing voice timbre conversion based on zero samples. It is mainly aimed at zero-sample singing voice timbre conversion scenarios, improves the song speech content encoding and speaker encoding in the feature decoupling stage, introduces song basic frequency encoding to characterize song rhythm to enhance feature representation, and improves timbre conversion quality through adversarial training tasks, thereby improving the generalization performance of timbre post-conversion.

[0008] To achieve the above-mentioned objectives, an embodiment provides a method for rapid conversion of singing voice timbre based on zero samples, comprising the following steps:

[0009] Construct a vocal dataset containing dry vocals and lyrics text;

[0010] A vocal feature decoupler is constructed that includes a Hubert model and a residual quantization codebook. The Hubert model is used to extract the initial content features of the dry vocals, and the residual quantization codebook is used to map the initial content features to an audio codebook index sequence. A text encoder is introduced to extract the morpheme index sequence and morpheme features of the lyrics text. The vocal feature decoupler is trained by cross-prediction based on the audio codebook index sequence, morpheme index sequence and morpheme features.

[0011] In the feature extractor, the trained singing feature decoupler is combined with audio super-resolution processing to extract the speech content features of the dry human voice, and at the same time, the pitch and timbre features of the dry human voice are extracted. In the generator, the synthesized human voice is generated based on the speech content features, pitch features, and timbre features. A discriminator is introduced to distinguish between the synthesized human voice and the dry human voice. The feature extractor, generator, and discriminator are trained using an adversarial training method.

[0012] When applying inference, the trained feature extractor and generator are used to generate a synthesized human voice as the converted singing voice, thus achieving rapid conversion of singing voice timbre.

[0013] Preferably, a text encoder is introduced to extract the morpheme index sequence and morpheme features of the lyrics text, including:

[0014] The text encoder includes a morpheme extraction unit and a morpheme feature extraction unit. The morpheme extraction unit extracts morphemes from the lyrics text and obtains the corresponding morpheme indexes based on the morpheme-index table, thereby forming a morpheme index sequence. The morpheme feature extraction unit extracts morpheme features based on the morpheme index sequence.

[0015] Preferably, the singing feature decoupler is trained based on cross-prediction using audio codebook index sequences, morpheme index sequences, and morpheme features, including:

[0016] A content predictor is introduced. The morpheme index sequence input to the content predictor is processed by embedding to obtain morpheme embedding features. The morpheme embedding features are fused with the morpheme features to obtain fused morpheme features. The audio codebook index sequence input to the content predictor is processed by embedding to obtain audio features. The fused morpheme features and audio features are cross-predicted, that is, the predicted audio codebook index sequence corresponding to speech is generated based on the fused morpheme features, and the predicted morpheme index sequence corresponding to text is generated based on the audio features.

[0017] An audio alignment loss is constructed based on the audio codebook index sequence and the predicted audio codebook index sequence, and a text alignment loss is constructed based on the morpheme index sequence and the predicted morpheme index sequence. The singing feature decoupler is trained based on the audio alignment loss and the morpheme alignment loss to optimize the parameters of the Hubert model and update the residual quantization codebook.

[0018] Preferably, the feature extractor utilizes a trained vocal feature decoupler combined with audio super-resolution processing to extract speech content features of the dry vocal voice, while simultaneously extracting pitch and timbre features of the dry vocal voice, including:

[0019] The trained singing feature decoupler is used to extract the audio codebook index sequence of the input dry human voice, and audio super-resolution processing is performed on the audio codebook index sequence to obtain the speech content features of the dry human voice.

[0020] Mel spectrum is extracted from dry vocals, and the fundamental frequency curve reflecting the pitch contour in the song is extracted from the Mel spectrum by the fundamental frequency extractor RMVPE as pitch feature. At the same time, timbre features are extracted from the Mel spectrum using a convolutional neural network.

[0021] Preferably, the synthesized human voice is generated in the generator based on speech content features, pitch features, and timbre features, including:

[0022] Speech content features, pitch features, and timbre features are all input into a multi-dimensional timbre embedding encoder. After each feature is unified in dimension through an embedding layer, any two features are cross-added through a cross-attention mechanism to obtain a speech embedding representation containing prosody, timbre, and content. Based on the speech embedding representation, a prior distribution representing the speech distribution is generated. When applying inference, the prior distribution and timbre features are processed by a flow model and then passed through an audio decoder to generate a synthesized human voice.

[0023] A posterior encoder is introduced to generate a posterior distribution of speech distribution based on the speech content features of the dry human voice. During adversarial training, the posterior distribution and timbre features are processed by a streaming model and then by an audio decoder to generate a synthesized human voice.

[0024] Preferably, when training the feature extractor, generator, and discriminator using adversarial training, the loss functions used include: adversarial loss of the adversarial network composed of the generator and discriminator, audio synthesis loss constructed based on the difference in Mel spectrum and feature map between the synthesized human voice and the original dry human voice, similarity loss based on the difference between the prior distribution and the posterior distribution, and quantization loss based on the residual quantization codebook.

[0025] Preferably, when applying inference, the speech content features and pitch features of the singing voice to be converted are extracted by a feature extractor, and the timbre features of the target human voice are directly extracted by a convolutional neural network. Then, a generator generates a synthesized human voice as the converted singing voice based on the speech content features and pitch features of the singing voice to be converted, as well as the timbre features of the target human voice.

[0026] To achieve the above-mentioned objectives, the embodiments also provide a zero-sample-based rapid vocal timbre conversion device, comprising:

[0027] The data construction module is used to build a vocal dataset containing dry vocals and lyrics text;

[0028] The pre-training module is used to construct a vocal feature decoupler that includes a Hubert model and a residual quantization codebook. The Hubert model is used to extract the initial content features of the dry vocals, and the residual quantization codebook is used to map the initial content features to an audio codebook index sequence. A text encoder is introduced to extract the morpheme index sequence and morpheme features of the lyrics text. The vocal feature decoupler is trained by cross-prediction based on the audio codebook index sequence, morpheme index sequence and morpheme features.

[0029] The adversarial training module is used in the feature extractor to extract speech content features of dry human voice by combining the trained singing feature decoupler with audio super-resolution processing. At the same time, it extracts the pitch and timbre features of the dry human voice. In the generator, a synthesized human voice is generated based on the speech content features, pitch features, and timbre features. A discriminator is introduced to distinguish between the synthesized human voice and the dry human voice. The feature extractor, generator, and discriminator are trained using an adversarial training method.

[0030] The application inference module is used to generate a synthesized human voice as a converted singing voice by utilizing the trained feature extractor and generator, thus achieving rapid conversion of singing voice timbre.

[0031] To achieve the above-mentioned objectives, the embodiments also provide a computing device, including a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, it is used to implement the above-mentioned zero-sample-based fast voice timbre conversion method.

[0032] To achieve the above-mentioned objectives, the embodiments also provide a computer-readable storage medium storing a program that, when executed by a processor, implements the above-mentioned method for rapid conversion of singing timbre based on zero samples.

[0033] Compared with the prior art, the beneficial effects of the present invention include at least the following:

[0034] A vocal feature decoupler incorporating a Hubert model and residual quantization codebook will be constructed to extract audio codebook index sequences. A text encoder will be introduced to extract morpheme features and morpheme index sequences. The vocal feature decoupler will be optimized through cross-prediction to improve the accuracy of speech content feature extraction. Based on this, pitch and timbre features representing prosody will be introduced. Adversarial training will be used to enhance the quality of synthesized human voices generated by the generator based on speech content features, pitch features, and timbre features. This will enable rapid conversion of song vocal timbre to user timbre while ensuring conversion quality. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1 This is a flowchart of the zero-sample-based fast vocal timbre conversion method provided in the embodiment;

[0037] Figure 2 This is a flowchart illustrating the construction process of the singing dataset provided in the embodiment;

[0038] Figure 3 This is a flowchart of the pre-training process of the singing feature decoupler provided in the embodiment;

[0039] Figure 4 This is a flowchart of the adversarial training provided in the embodiment;

[0040] Figure 5 This is an application reasoning flowchart provided in the embodiment;

[0041] Figure 6 This is a schematic diagram of the structure of the zero-sample-based rapid voice timbre conversion device provided in the embodiment;

[0042] Figure 7 This is a schematic diagram of the computing device provided in the embodiment. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the scope of protection of this invention.

[0044] While voice timbre conversion methods can be implemented by drawing on traditional speech synthesis and conversion techniques, there is currently a lack of large-scale integrated datasets of raw human voice data for training models. Furthermore, these methods require anywhere from ten minutes to several hours of user data to fine-tune the models. For a single ordinary user, this scale of data is difficult to achieve, thus significantly limiting the widespread application of voice timbre conversion.

[0045] To address the technical challenges of missing datasets and the reliance on few-sample data for model fine-tuning, this invention provides a zero-sample-based rapid vocal timbre conversion scheme. It aggregates and processes vocal data to construct a vocal dataset, decouples vocal audio features based on the differences between vocal and speech, improves the accuracy of feature decoupling using auxiliary tasks, and trains the model through adversarial training. This decouples and combines the features of the original vocal to be converted and the target user's voice to generate the timbre-converted vocal, ultimately achieving zero-sample rapid vocal timbre conversion and reducing the user's required reference audio data duration to five seconds.

[0046] like Figure 1 As shown in the embodiment, a method for fast conversion of singing voice timbre based on zero samples is provided, including the following steps:

[0047] S1, construct a vocal dataset containing dry vocals and lyrics text.

[0048] In the embodiments, such as Figure 2 As shown, hundreds of thousands of song data points covering multiple languages ​​were obtained through web scraping and compiling open-source datasets. To eliminate interference from factors other than human voices, the Demucus voice classifier was used to separate the song audio to obtain the pure human voice, i.e., dry human voice. Furthermore, to obtain the lyrics from the songs and improve the accuracy of the model's speech content extraction, complete lyrics and the start and end times of pronunciation for each character were extracted from Chinese and foreign language songs using Funasr and Whisper v3 respectively. Finally, the data underwent manual screening to remove all errors in the recognition results, resulting in a song dataset containing dry human voice and lyrics.

[0049] S2, construct a vocal feature decoupler and pre-train it.

[0050] In the embodiments, such as Figure 3As shown, the constructed vocal feature decoupling mechanism includes a Hubert model and a residual quantization codebook. The Hubert model is pre-trained and used to extract initial content features from the dry vocal audio. Specifically, in the Hubert model, audio temporal features are obtained from the dry vocal audio file at a sampling rate of 16kHz, and Mel spectral features are extracted through Fast Fourier Transform and Mel filter. The initial content features are then obtained based on the Mel spectral features. The residual quantization codebook is composed of several sequentially connected quantization codebooks. Each quantization codebook transforms the input vector into a corresponding quantization codebook vector index sequence, and outputs the result of subtracting the input vector from the corresponding codebook vector to the next quantization codebook. This process is repeated until the final result is formed by concatenating the vector index sequences obtained from all codebooks. By applying the residual quantization codebook, the initial content features can be mapped to an audio codebook index sequence, thus enabling feature extraction from the perspective of speech content.

[0051] In this embodiment, feature extraction is also performed on the text content. Specifically, a text encoder is introduced to extract the morpheme index sequence and morpheme features from the lyrics text. The text encoder includes a morpheme extraction unit and a morpheme feature extraction unit. The morpheme extraction unit extracts morphemes from the lyrics text. Specifically, based on language differences, pypinyin, g2p_en, and pyopenjtalk are used to convert each character in Chinese, English, and Japanese into its corresponding smallest phonetic unit—a morpheme. Based on the morpheme-index table, the morpheme index corresponding to each morpheme is obtained, thus forming a morpheme index sequence. To address the sparsity problem of the index, the morpheme feature extraction unit extracts morpheme features based on the morpheme index sequence. Specifically, the BERT text processing model, which is pre-trained on a large scale, is used to perform feature embedding processing on the morpheme index sequence, ultimately obtaining high-dimensional morpheme features.

[0052] Due to the domain differences between the extracted audio and text, the extracted morpheme index sequences and audio codebook index sequences still need to be aligned through training. This alignment process ensures that the extracted audio codebook index sequences accurately reflect the text content contained in the morpheme features and reduces interference information such as timbre and prosody included in the Hubert model. Specifically, a content predictor is introduced to assist feature alignment through a speech content prediction task. The audio codebook index sequence, morpheme index sequence, and morpheme features are all input into the content predictor for cross-prediction to train the singing feature decoupling device, enabling the corresponding features of the two sequences to be aligned and reflect the original content information.

[0053] Specifically, the content predictor adopts a Decoder structure based on Transformer. The morpheme index sequence input to the content predictor is processed by embedding to obtain morpheme embedding features. These morpheme embedding features are fused with the morpheme features to obtain fused morpheme features. The audio codebook index sequence input to the content predictor is processed by embedding to obtain audio features. The fused morpheme features and audio features are cross-predicted, that is, the predicted audio codebook index sequence corresponding to speech is generated based on the fused morpheme features, and the predicted morpheme index sequence corresponding to text is generated based on the audio features.

[0054] During training, the corresponding audio segments are first read from the time intervals of the lyrics annotations in the singing dataset to obtain paired lyrics text and dry human voice as speech information. Then, the audio codebook index sequence, morpheme index sequence, and morpheme features are extracted through the singing feature decoupler and text encoder. Next, the predicted audio codebook index sequence and predicted morpheme index sequence are generated through the content predictor. Finally, an audio alignment loss is constructed based on the audio codebook index sequence and the predicted audio codebook index sequence, and a text alignment loss is constructed based on the morpheme index sequence and the predicted morpheme index sequence. The singing feature decoupler is trained based on the audio alignment loss and the morpheme alignment loss to optimize the parameters of the Hubert model and update the residual quantization codebook. This ensures the accuracy of content feature decoupling, thereby ensuring the accuracy of subsequent speech content feature extraction.

[0055] S3 extracts the speech content features of the dry human voice based on the trained singing feature decoupler. At the same time, it extracts the pitch and timbre features of the dry human voice and then performs adversarial training based on the generator and discriminator.

[0056] In this embodiment, a feature extractor is constructed based on the trained vocal feature decoupler. The feature extractor includes the trained vocal feature decoupler, audio super-resolution processing, a fundamental frequency extractor (RMVPE), and a convolutional neural network. The convolutional neural network includes fully connected layers, convolutional layers, and self-attention layers. Specifically, as shown... Figure 4 As shown, the process of extracting features using a feature extractor includes:

[0057] The trained singing feature decoupler is used to extract the audio codebook index sequence of the input dry human voice, and audio super-resolution processing is performed on the audio codebook index sequence to obtain the speech content features of the dry human voice.

[0058] Mel spectrum is extracted from dry vocals. To enhance the rhythm of the song, a fundamental frequency curve reflecting the pitch contours of the song is extracted from the Mel spectrum using the fundamental frequency extractor RMVPE, serving as the pitch feature.

[0059] Since Mel spectrum features can well reflect human ear's perception of audio, convolutional neural networks are used to extract timbre features from Mel spectrum. That is, after mapping Mel spectrum using fully connected layers, convolution is performed through convolutional layers, and then timbre features are obtained by focusing on the features obtained from convolution through self-attention layers.

[0060] The aforementioned feature extractor has been used to accurately extract speech content features, pitch features, and timbre features from speech. The extracted timbre features are used to form the speaker coding features. Subsequently, an adversarial training strategy is introduced to jointly generate and discriminate the generator and discriminator to complete the singing voice synthesis after timbre conversion. The specific process is as follows: Figure 4 As shown:

[0061] In the generator, speech content features, along with pitch and timbre features, serve as control conditions for synthesized human voice. These features are input to a multi-dimensional timbre embedding encoder, where each feature is unified through an embedding layer. Then, any two features are cross-added using a cross-attention mechanism to obtain a speech embedding representation that encapsulates prosody, timbre, and content. Based on this embedding representation, a prior distribution representing the speech distribution is generated, represented by the mean and log-variance. Simultaneously, a posterior encoder is introduced to generate a posterior distribution of the speech distribution based on the speech content features of the raw human voice. This posterior distribution is also represented by the mean and log-variance. During adversarial training, the posterior distribution and timbre features are processed by a flow-based model before being passed to an audio decoder to generate the synthesized human voice. The flow model can employ the Glow model, and the audio decoder uses a deconvolutional structure.

[0062] During adversarial training, the discriminator uses stacked one-dimensional and two-dimensional convolutional networks to comprehensively evaluate the spatiotemporal features of the audio, and then distinguishes between the audio of synthesized human voice and the audio of original dry human voice. The loss function employed includes not only the adversarial loss of the adversarial network composed of the generator and discriminator, but also various other losses corresponding to the generator. Specifically, it includes an audio synthesis loss constructed based on the Mel-spectral difference and feature map difference between the synthesized human voice and the original dry human voice. The Mel-spectral difference and feature map difference can both use LI loss or L2 loss. The feature map loss refers to the feature maps extracted from the synthesized human voice and the original dry human voice. These feature maps can come from the intermediate feature maps extracted by the discriminator, or feature maps extracted by other network structures such as VGG. It also includes a similarity loss based on the prior distribution and the posterior distribution. This similarity loss can be the KL divergence loss, specifically calculated using the mean and log-variance of the prior distribution and the mean and corresponding variance of the posterior distribution. It also includes a quantization loss based on the residual quantization codebook. The residual quantization codebook is continuously optimized during training, and the difference between the features before and after mapping by the residual quantization codebook is used as the quantization loss.

[0063] By using the loss function constructed above to perform adversarial training on the feature extractor, generator, and discriminator, the synthesized human voice audio generated by the generator becomes increasingly difficult to distinguish, and the prior distribution of the prior space and the posterior distribution of the posterior space gradually approach each other, ensuring that in subsequent inference applications, the timbre-converted song audio can be directly synthesized based on the prior original song audio and the user's target timbre reference audio.

[0064] S4 utilizes the trained feature extractor and generator to generate a synthesized human voice as the converted singing voice during inference, achieving rapid conversion of singing voice timbre.

[0065] In this embodiment, during model inference, the decoupling results of the user-uploaded speech features and the decoupling results of the original song are fused. The timbre features of the user's speech replace the timbre features of the original song, generating the final synthesized human voice and achieving song timbre conversion. Specifically, after completing the above training, the inference process for song timbre conversion no longer requires a posterior encoder, but directly uses the trained feature extractor and generator to generate the synthesized human voice, such as... Figure 4 As shown, the process includes the following:

[0066] First, the speech content features and pitch features of the singing voice to be converted are extracted by a feature extractor. Specifically, the audio codebook index sequence of the input singing voice to be converted is extracted by using a trained singing voice feature decoupler, and audio super-resolution processing is performed on the audio codebook index sequence to obtain the speech content features of the singing voice to be converted. At the same time, a convolutional neural network is used to directly extract the timbre features of the target human voice, that is, the timbre features of the target human voice are used to replace the timbre features of the singing voice to be converted.

[0067] Then, a synthesized human voice is generated based on the speech content features and pitch features of the human voice to be converted, as well as the timbre features of the target human voice, to generate the converted singing voice. Specifically, the speech content features and pitch features of the human voice to be converted, and the timbre features of the target human voice are all input into a multi-dimensional timbre embedding encoder to generate a prior distribution representing the speech distribution. This prior distribution and timbre features are processed by a streaming model and then passed through an audio decoder to generate the synthesized human voice as the converted singing voice.

[0068] In the above conversion process, the target human voice is an audio that did not appear during the training process, which is a true zero sample. Based on the target human voice, the timbre of the human voice to be converted is converted. This solves the problem that current singing voice timbre conversion applications still require tens of minutes to several hours of data for few-sample learning. It enables the model to convert the timbre of a song's human voice into the user's timbre based on about five seconds of user recording, which greatly reduces the usage threshold of singing voice timbre conversion and is of great significance for the large-scale promotion of singing voice timbre conversion applications.

[0069] like Figure 6As shown, the embodiment also provides a zero-sample-based fast vocal timbre conversion device 60, including a data construction module 61, a pre-training module 62, an adversarial training module 63, and an application inference module 64, wherein the data construction module 61 is used to construct a vocal dataset containing dry vocals and lyrics text.

[0070] The pre-training module 62 is used to construct and pre-train a vocal feature decoupler. Specifically, it constructs a vocal feature decoupler that includes a Hubert model and a residual quantization codebook. The Hubert model is used to extract the initial content features of the dry vocals, and the residual quantization codebook is used to map the initial content features to an audio codebook index sequence. A text encoder is introduced to extract the morpheme index sequence and morpheme features of the lyrics text. The vocal feature decoupler is trained by cross-prediction based on the audio codebook index sequence, the morpheme index sequence, and the morpheme features.

[0071] The adversarial training module 63 is used to extract speech content features of the dry human voice based on the trained singing feature decoupler, and simultaneously extract the pitch and timbre features of the dry human voice, and then perform adversarial training based on the generator and discriminator. Specifically, it is used in the feature extractor to extract speech content features of the dry human voice by combining the trained singing feature decoupler with audio super-resolution processing, and simultaneously extract the pitch and timbre features of the dry human voice. In the generator, a synthesized human voice is generated based on the speech content features, pitch features, and timbre features. A discriminator is introduced to distinguish between the synthesized human voice and the dry human voice, and an adversarial training method is used to train the feature extractor, generator, and discriminator.

[0072] The application inference module 64 is used to generate a synthesized human voice as a converted singing voice by utilizing the trained feature extractor and generator, thereby achieving rapid conversion of singing voice timbre.

[0073] It should be noted that the zero-sample-based rapid voice timbre conversion device provided in the above embodiments should be illustrated using the above-described functional module division as an example when performing rapid voice timbre conversion. The functions can be assigned to different functional modules as needed, i.e., the internal structure of the terminal or server can be divided into different functional modules to complete all or part of the functions described above. Furthermore, the zero-sample-based rapid voice timbre conversion device and the zero-sample-based rapid voice timbre conversion construction method embodiments provided in the above embodiments belong to the same concept. For details of their specific implementation process, please refer to the zero-sample-based rapid voice timbre conversion method embodiments, which will not be repeated here.

[0074] Based on the same inventive concept, the embodiment also provides a computing device, including a memory and one or more processors. The memory stores executable code, and when the one or more processors execute the executable code, it is used to implement the above-described zero-sample-based fast vocal timbre conversion method, specifically including the following steps:

[0075] S1, Construct a vocal dataset containing dry vocals and lyrics text;

[0076] S2, Construct a singing feature decoupler and pre-train it;

[0077] S3 extracts speech content features of dry human voice based on the trained singing feature decoupler. At the same time, it extracts pitch and timbre features of dry human voice and then performs adversarial training based on the generator and discriminator.

[0078] S4 utilizes the trained feature extractor and generator to generate a synthesized human voice as the converted singing voice during inference, achieving rapid conversion of singing voice timbre.

[0079] like Figure 7 As shown, the computing device provided in this embodiment, at the hardware level, includes not only a processor and memory, but also internal buses, network interfaces, memory, and other hardware required for business operations. The memory is non-volatile memory. The processor reads the corresponding computer program from the non-volatile memory into memory and then runs it to implement the zero-sample-based fast voice timbre conversion method described in S1-S4 above. Of course, besides software implementation, this invention does not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution entity of the following processing flow is not limited to individual logic units, but can also be hardware or logic devices.

[0080] Based on the same inventive concept, the embodiments also provide a computer-readable storage medium storing a program thereon, which, when executed by a processor, implements the above-described zero-sample-based rapid vocal timbre conversion method, specifically including the following steps:

[0081] S1, Construct a vocal dataset containing dry vocals and lyrics text;

[0082] S2, Construct a singing feature decoupler and pre-train it;

[0083] S3 extracts speech content features of dry human voice based on the trained singing feature decoupler. At the same time, it extracts pitch and timbre features of dry human voice and then performs adversarial training based on the generator and discriminator.

[0084] S4 utilizes the trained feature extractor and generator to generate a synthesized human voice as the converted singing voice during inference, achieving rapid conversion of singing voice timbre.

[0085] In this embodiment, the computer-readable medium includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. The information can be computer-readable instructions, data structures, program modules, or other data.

[0086] The specific embodiments described above illustrate the technical solution and beneficial effects of the present invention in detail. It should be understood that the above description is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A zero-shot based song timbre fast conversion method, characterized in that, Includes the following steps: Construct a vocal dataset containing dry vocals and lyrics text; A vocal feature decoupler is constructed, comprising a Hubert model and a residual quantization codebook. The Hubert model extracts initial content features from the dry vocal text, while the residual quantization codebook maps these initial content features to an audio codebook index sequence. A text encoder is introduced to extract morpheme index sequences and morpheme features from the lyrics text. The vocal feature decoupler is trained by cross-prediction based on the audio codebook index sequence, morpheme index sequence, and morpheme features. Specifically, this includes introducing a content predictor, embedding the morpheme index sequence input to the content predictor to obtain morpheme embedding features, and fusing these morpheme embedding features with the morpheme features to obtain the fused morpheme embedding features. The audio codebook index sequence input to the content predictor is embedded to obtain audio features. The fused morpheme features and audio features are cross-predicted, that is, the fused morpheme features are used to generate the predicted audio codebook index sequence for speech, and the audio features are used to generate the predicted morpheme index sequence for text. Based on the audio codebook index sequence and the predicted audio codebook index sequence, an audio alignment loss is constructed, and based on the morpheme index sequence and the predicted morpheme index sequence, a text alignment loss is constructed. Based on the audio alignment loss and the morpheme alignment loss, the singing feature decoupler is trained to optimize the parameters of the Hubert model and update the residual quantization codebook. In the feature extractor, a trained vocal feature decoupler combined with audio super-resolution processing is used to extract speech content features of the dry human voice, as well as pitch and timbre features. In the generator, a synthesized human voice is generated based on the speech content features, pitch features, and timbre features. Specifically, the speech content features, pitch features, and timbre features are all input into a multi-dimensional timbre embedding encoder. After each feature is unified through an embedding layer, any two features are cross-added using a cross-attention mechanism to obtain a speech embedding representation containing prosody, timbre, and content. A prior distribution representing the speech distribution is generated based on the speech embedding representation. During inference, the prior distribution and timbre features are processed by a stream model and then by an audio decoder to generate the synthesized human voice. A posterior encoder is introduced to generate a posterior distribution of the speech distribution based on the speech content features of the dry human voice. During adversarial training, the posterior distribution and timbre features are processed by a stream model and then by an audio decoder to generate the synthesized human voice. A discriminator is introduced to distinguish between the synthesized human voice and the dry human voice, and the feature extractor, generator, and discriminator are trained using an adversarial training method. When applying inference, the trained feature extractor and generator are used to generate a synthesized human voice as the converted singing voice, thus achieving rapid conversion of singing voice timbre.

2. The zero-sample based vocal timbre quick conversion method according to claim 1, wherein, A text encoder is introduced to extract the morpheme index sequence and morpheme features of the lyrics text, including: The text encoder includes a morpheme extraction unit and a morpheme feature extraction unit. The morpheme extraction unit extracts morphemes from the lyrics text and obtains the corresponding morpheme indexes based on the morpheme-index table, thereby forming a morpheme index sequence. The morpheme feature extraction unit extracts morpheme features based on the morpheme index sequence.

3. The zero-sample based vocal timbre quick conversion method according to claim 1, wherein, The feature extractor utilizes a trained vocal feature decoupler combined with audio super-resolution processing to extract speech content features from the dry vocal voice, simultaneously extracting pitch and timbre features, including: The trained singing feature decoupler is used to extract the audio codebook index sequence of the input dry human voice, and audio super-resolution processing is performed on the audio codebook index sequence to obtain the speech content features of the dry human voice. Mel spectrum is extracted from dry vocals, and the fundamental frequency curve reflecting the pitch contour in the song is extracted from the Mel spectrum by the fundamental frequency extractor RMVPE as pitch feature. At the same time, timbre features are extracted from the Mel spectrum using a convolutional neural network.

4. The zero-sample based vocal timbre quick conversion method according to claim 1, wherein, When training the feature extractor, generator, and discriminator using adversarial training, the loss functions used include: adversarial loss of the adversarial network composed of the generator and discriminator, audio synthesis loss based on the difference in Mel spectrum and feature map between the synthesized human voice and the original dry human voice, similarity loss based on the difference between the prior distribution and the posterior distribution, and quantization loss based on the residual quantization codebook.

5. The method for rapid conversion of singing timbre based on zero samples according to claim 3, characterized in that, When applying inference, the speech content features and pitch features of the singing voice to be converted are extracted by a feature extractor. At the same time, a convolutional neural network is used to directly extract the timbre features of the target human voice. Then, a generator is used to generate a synthesized human voice as the converted singing voice based on the speech content features and pitch features of the singing voice to be converted, as well as the timbre features of the target human voice.

6. A zero-sample-based rapid voice timbre conversion device, characterized in that, include: The data construction module is used to build a vocal dataset containing dry vocals and lyrics text; The pre-training module is used to construct a vocal feature decoupler containing a Hubert model and a residual quantization codebook. The Hubert model extracts initial content features from the dry vocal text, and the residual quantization codebook maps these initial content features to an audio codebook index sequence. A text encoder is introduced to extract morpheme index sequences and morpheme features from the lyrics text. The vocal feature decoupler is trained by cross-prediction based on the audio codebook index sequence, morpheme index sequence, and morpheme features. Specifically, this includes introducing a content predictor, embedding the morpheme index sequence input to the content predictor to obtain morpheme embedding features, and fusing these morpheme embedding features with the morpheme features to obtain a fused morpheme feature. The fused morpheme features are used to obtain audio features from the audio codebook index sequence input to the content predictor. The fused morpheme features and audio features are then cross-predicted, i.e., a predicted audio codebook index sequence for speech is generated based on the fused morpheme features, and a predicted morpheme index sequence for text is generated based on the audio features. An audio alignment loss is constructed based on the audio codebook index sequence and the predicted audio codebook index sequence, and a text alignment loss is constructed based on the morpheme index sequence and the predicted morpheme index sequence. The singing feature decoupler is trained based on the audio alignment loss and the morpheme alignment loss to optimize the parameters of the Hubert model and update the residual quantization codebook. The adversarial training module is used in the feature extractor to extract speech content features of the dry human voice by combining a trained singing feature decoupler with audio super-resolution processing. Simultaneously, it extracts the pitch and timbre features of the dry human voice. In the generator, a synthesized human voice is generated based on the speech content features, pitch features, and timbre features. Specifically, the speech content features, pitch features, and timbre features are all input into a multi-dimensional timbre embedding encoder. After each feature is unified in dimension through an embedding layer, any two features are cross-added using a cross-attention mechanism to obtain a result containing prosody, timbre, and inner... The system constructs a speech embedding representation and generates a prior distribution representing the speech distribution based on the speech embedding representation. During inference, the prior distribution and timbre features are processed by a streaming model and then passed through an audio decoder to generate a synthesized human voice. A posterior encoder is introduced to generate a posterior distribution of the speech distribution based on the speech content features of the dry human voice. During adversarial training, the posterior distribution and timbre features are processed by a streaming model and then passed through an audio decoder to generate a synthesized human voice. A discriminator is introduced to distinguish between the synthesized human voice and the dry human voice, and an adversarial training method is used to train the feature extractor, generator, and discriminator. The application inference module is used to generate a synthesized human voice as a converted singing voice by utilizing the trained feature extractor and generator, thus achieving rapid conversion of singing voice timbre.

7. A computing device comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that, When the one or more processors execute the executable code, they are used to implement the zero-sample-based fast vocal timbre conversion method according to any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, It stores a program that, when executed by a processor, implements the zero-sample-based rapid vocal timbre conversion method according to any one of claims 1-5.

Images