Training methods, audio generation methods, and related devices for a singing voice conversion system

Abstract

This invention provides a training method for a vocal conversion system, a method for generating audio based on the vocal conversion system, and related apparatus, for improving the similarity between the timbre of the synthesized vocal and the timbre of the vocal to be converted. The method includes: acquiring multiple reference audios of a first target timbre; inputting the multiple reference audios into a timbre encoder to obtain a timbre encoding vector; inputting the posterior probability of the phonemes and the fundamental frequency of the vocal to be converted into a text encoder to obtain prior distribution parameters of the vocal content; sampling according to the prior distribution parameters to obtain a text sampling value vector of the vocal content to be converted; inputting the text sampling value vector and the timbre encoding vector into a timbre-aware attention mechanism module to determine a new timbre encoding vector; using the new timbre encoding vector as a new input added to the vocal conversion system, calculating the reconstruction loss of the vocal conversion system, and training the vocal conversion system based on the reconstruction loss.

Description

Technical Field

[0001] This invention relates to the field of audio data processing technology, and in particular to a method for converting samples in a singing voice conversion system, a method for generating audio based on a singing voice conversion system, and related apparatus. Background Technology

[0002] Vocal conversion refers to a technique that, when inputting a vocal audio clip to be converted and a target audio timbre, can convert the vocal audio clip's timbre to the target timbre while preserving the vocal content. This vocal conversion system can help people easily synthesize any song with a specified timbre for use in the audio editing functions of karaoke software.

[0003] Existing technologies for voice conversion generally use zero-shot voice conversion systems (Zero-shotSVC). Such zero-shot voice conversion systems refer to systems that do not use target timbre data to train the voice conversion system, but directly input the target timbre data during the inference stage, thereby converting the timbre of the voice to be converted.

[0004] Zero-sample vocal conversion systems, because they do not fine-tune the vocal conversion model before using it, have a low similarity between the timbre of the synthesized audio obtained by using a zero-sample vocal conversion system and the target timbre of the singer to be converted. Summary of the Invention

[0005] This invention provides a training method for a singing voice conversion system, a method for generating audio based on the singing voice conversion system, and related devices. The singing voice system incorporates a timbre encoder and a timbre-aware attention mechanism module. The timbre encoder inputs multiple reference audio tracks representing a first target timbre. These reference audio tracks contain multiple audio recordings of the singer whose voice is to be converted, each in a different vocal register. Because the resonance methods used in different vocal registers differ, the timbre of the singer's voice varies across the multiple reference audio tracks. This allows the singing voice conversion system, after converting the sample, to learn a target timbre suitable for the voice to be converted from the multiple reference audio tracks based on the singing content representation vector of the voice to be converted. Furthermore, based on the target timbre suitable for the voice to be converted, the system synthesizes the timbre of the voice to be converted, thereby improving the similarity between the synthesized timbre and the timbre of the voice to be converted.

[0006] A first aspect of this application provides a training method for a singing voice conversion system, the singing voice conversion system including at least a text encoder, a timbre encoder, and a timbre perception attention mechanism module, the method comprising:

[0007] Obtain multiple reference audios of the first target timbre, wherein the multiple reference audios include at least multiple audios of the singer whose voice is to be converted in different vocal registers;

[0008] The multiple reference audios are input to the timbre encoder to perform timbre encoding on the multiple reference audios, thereby obtaining the timbre encoding vector of the multiple reference audios output by the timbre encoder;

[0009] The PPG phoneme posterior probability and the pitch fundamental frequency of the singing voice to be converted are input into the text encoder to obtain the prior distribution parameters of the singing voice content to be converted output by the text encoder.

[0010] Based on the prior distribution parameters, samples are taken from the posterior probability of the PPG phonemes of the vocal to be converted and the fundamental frequency of the pitch of the vocal to be converted to obtain the text sample value vector of the vocal content to be converted.

[0011] The text sampling value vector of the singing content to be converted and the timbre encoding vector of the multiple reference audios are input into the timbre perception attention mechanism module to determine a new timbre encoding vector based on the timbre encoding vector of the multiple reference audios and the text sampling value vector of the singing content to be converted.

[0012] The new timbre encoding vector is used as a new input added to the singing voice conversion system to calculate the reconstruction loss of the singing voice conversion system;

[0013] The singing conversion system is trained using reconstruction loss and backpropagation algorithm.

[0014] A second aspect of this application provides a method for generating audio based on a vocal conversion system, wherein the vocal conversion system includes at least a text encoder, a timbre encoder, a timbre-aware attention mechanism module, and a decoder, and the method includes:

[0015] Obtain multiple reference audios for the second target timbre, wherein the multiple reference audios for the second target timbre include at least multiple audios from different singers in different vocal registers;

[0016] The multiple reference audios are input to the timbre encoder to perform timbre encoding on the multiple reference audios, thereby obtaining the timbre encoding vector of the multiple reference audios output by the timbre encoder;

[0017] The PPG phoneme posterior probability and the pitch fundamental frequency of the singing voice to be converted are input into the text encoder to obtain the prior distribution parameters of the singing voice content to be converted output by the text encoder.

[0018] Based on the prior distribution parameters, samples are taken from the posterior probability of the PPG phonemes of the vocal to be converted and the fundamental frequency of the pitch of the vocal to be converted to obtain the text sample value vector of the vocal content to be converted.

[0019] The text sampling value vector of the singing content to be converted and the timbre encoding vector of the multiple reference audios are input into the timbre perception attention mechanism module to determine a new timbre encoding vector based on the timbre encoding vector of the multiple reference audios and the text sampling value vector of the singing content to be converted.

[0020] The text sample value vector of the singing content to be converted and the new timbre encoding vector are used as inputs to the decoder to obtain the synthesized singing audio.

[0021] A third aspect of this application provides a computer device including a processor. When the processor executes a computer program stored in a memory, it is used to implement the training method of the singing voice conversion system provided in the first aspect of this application, or the method of generating audio based on the singing voice conversion system provided in the second aspect of this application.

[0022] The fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it is used to implement the training method of the singing voice conversion system provided in the first aspect of this application, or the method of generating audio based on the singing voice conversion system provided in the second aspect of this application.

[0023] The fifth aspect of this application provides a computer program product having a computer program stored thereon, characterized in that, when the computer program is executed by a processor, it is used to implement the training method of the singing voice conversion system provided in the first aspect of this application, or the method of generating audio based on the singing voice conversion system provided in the second aspect of this application.

[0024] As can be seen from the above technical solutions, the embodiments of the present invention have the following advantages:

[0025] This application provides a training method for a singing voice conversion system. The singing voice conversion system includes at least a text encoder, a timbre encoder, and a timbre-aware attention mechanism module. The method includes: acquiring multiple reference audios of a first target timbre, wherein the multiple reference audios include at least multiple audios of the singer whose voice to be converted is in different vocal registers; inputting the multiple reference audios into the timbre encoder to perform timbre encoding on the multiple reference audios, obtaining a timbre encoding vector of the multiple reference audios output by the timbre encoder; and inputting the PPG phoneme posterior probability of the voice to be converted and the pitch fundamental frequency of the voice to be converted into the text encoder to obtain the timbre encoding vector of the voice to be converted output by the text encoder. The prior distribution parameters of the content are determined; based on the prior distribution parameters, samples are taken from the posterior probability of the PPG phonemes of the vocal to be converted and the fundamental frequency of the pitch of the vocal to be converted to obtain the text sample value vector of the vocal content to be converted; the text sample value vector of the vocal content to be converted and the timbre encoding vectors of multiple reference audios are input into the timbre perception attention mechanism module to determine a new timbre encoding vector based on the timbre encoding vectors of multiple reference audios and the text sample value vector of the vocal content to be converted; the new timbre encoding vector is used as a new input added to the vocal conversion system, the reconstruction loss of the vocal conversion system is calculated, and the vocal conversion system is trained based on the reconstruction loss and the backpropagation algorithm.

[0026] Because this application embodiment adds a timbre encoder and a timbre perception attention mechanism module to the singing conversion system, and inputs multiple reference audios of the first target timbre into the timbre encoder, and the multiple reference audios include at least multiple audios of the singer to be converted in different vocal registers. Since the resonance methods used in singing high and low registers are different, the timbre of the singer to be converted in the multiple reference audios is also different. Thus, the singing conversion system after conversion can learn the target timbre that is suitable for the singing to be converted from the multiple reference audios according to the singing content representation vector of the singing to be converted. Then, based on the target timbre that is suitable for the singing to be converted, the timbre of the singing to be converted is synthesized, thereby improving the similarity between the timbre of the synthesized singing and the timbre of the singing to be converted. Attached Figure Description

[0027] Figure 1 This is a schematic diagram of the system architecture of the vocal training conversion system provided in the embodiments of this application;

[0028] Figure 2 A schematic diagram of an embodiment of the training method for the singing voice conversion system in this application;

[0029] Figure 3 This is a schematic diagram of the architecture of the singing voice conversion system in the embodiments of this application;

[0030] Figure 4This is a schematic diagram of the architecture of the singing voice conversion system based on the VITS variational autoencoder in the embodiments of this application;

[0031] Figure 5 This is a schematic diagram of the training process of the singing conversion system based on the VITS variational autoencoder in the embodiments of this application;

[0032] Figure 6 This is another schematic diagram of the singing voice conversion system based on the VITS variational autoencoder in the embodiments of this application;

[0033] Figure 7 This is a schematic diagram of another training process for the singing conversion system based on the VITS variational autoencoder in an embodiment of this application;

[0034] Figure 8 This is a schematic diagram of the architecture of the Fastspeeh fast speech synthesis model in the embodiments of this application;

[0035] Figure 9 This is a schematic diagram illustrating the training process of the Fastspeeh-based fast speech synthesis model in an embodiment of this application.

[0036] Figure 10 Another architectural diagram of the Fastspeeh fast speech synthesis model in this application embodiment;

[0037] Figure 11 This is a schematic diagram illustrating another training process for the Fastspeeh-based fast speech synthesis model in an embodiment of this application;

[0038] Figure 12 This is a schematic diagram of another model architecture of the Diffusion-based generation model in the embodiments of this application;

[0039] Figure 13 This is a schematic diagram illustrating another training process for the Diffusion-based generative model in an embodiment of this application.

[0040] Figure 14 This is a schematic diagram of an embodiment of the method for generating audio based on a singing voice conversion system in this application.

[0041] Figure 15 This is a schematic diagram of the architecture of the singing voice conversion system in the embodiments of this application;

[0042] Figure 16 This is a schematic diagram of the architecture of the singing voice conversion system based on the VITS variational autoencoder in the embodiments of this application;

[0043] Figure 17 This is a schematic diagram of an embodiment of the method for generating audio using a singing voice conversion system based on the VITS variational autoencoder in this application.

[0044] Figure 18 This is a schematic diagram of an embodiment of a method for generating audio based on the Fastspeeh fast speech synthesis model or the Diffusion diffusion generation model in this application. Detailed Implementation

[0045] This invention provides a training method for a vocal conversion system, a method for generating audio based on the vocal conversion system, and related devices, which are used to improve the similarity between the timbre of the synthesized vocal and the timbre of the vocal to be converted.

[0046] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0047] The terms "first," "second," "third," "fourth," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0048] This application provides a first aspect: a training method for a singing voice conversion system. This method adds a timbre encoder and a timbre-aware attention mechanism module to the existing singing voice conversion system. The general principle of this training method is as follows: multiple reference audio samples of a first target timbre are input to the timbre encoder to perform timbre encoding on the multiple reference audio samples, obtaining a timbre encoding vector output by the timbre encoder for the multiple reference audio samples. The multiple reference audio samples include at least multiple audio samples of the singer to be converted in different vocal registers. Then, the PPG phoneme posterior probability of the singing voice to be converted and the pitch fundamental frequency of the singing voice to be converted are input to the text encoder in the singing voice conversion system to obtain the output of the text encoder. The prior distribution parameters of the vocal content to be converted are used to sample the posterior probabilities of the PPG phonemes and the fundamental frequency of the pitch of the vocal content to be converted, so as to obtain the text sample value vector of the vocal content to be converted. Finally, the text sample value vector of the vocal content to be converted and the timbre encoding vectors of multiple reference audios are input into the timbre perception attention mechanism module to determine a new timbre encoding vector based on the timbre encoding vectors of multiple reference audios and the text sample value vector of the vocal content to be converted. The new timbre encoding vector is used as a new input added to the vocal conversion system. The reconstruction loss of the vocal conversion system is calculated. Then, the vocal conversion system is trained according to the reconstruction loss and the backpropagation algorithm. The training method for the singing voice conversion system provided in this application embodiment adds a timbre encoder and a timbre perception attention mechanism module to the singing voice conversion system. This allows the timbre encoder and timbre perception attention mechanism module to determine a new timbre encoding vector based on the timbre encoding vectors of multiple reference audios and the text sampling value vector of the singing voice content to be converted. The new timbre encoding vector is then used as a new input to train the singing voice conversion system. As a result, the trained singing voice conversion system can learn a target timbre that is suitable for the singing voice content to be converted based on the text sampling value vector of the singing voice content to be converted and multiple reference audios. Then, based on the target timbre that is suitable for the singing voice content to be converted, the system synthesizes the singer's singing voice content, thereby improving the similarity between the timbre of the synthesized singing voice and the singer's singing voice timbre.

[0049] To better implement the training method for the aforementioned singing voice conversion system, this application provides a system for training a singing voice conversion system. Please refer to [link to relevant documentation]. Figure 1 , Figure 1This is a schematic diagram of the system architecture of the vocal training conversion system provided in this application embodiment. The system may include at least one terminal device 101 and a server 102. Different types of applications may be installed on the terminal device 101, such as karaoke applications, instant messaging applications, live streaming applications, conference communication applications, etc. The terminal device 101 may be a smartphone, tablet, laptop, desktop computer, smart vehicle, etc. The server 102 may be used to store application data and audio data generated by different types of applications on the terminal device 101. The server 102 may be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, etc.

[0050] The training method for the aforementioned vocal conversion system is executed by either terminal device 101 or server 102. When the training method is executed by terminal device 101, the audio data generated by terminal device 101 in different types of applications can be stored on the server. When terminal device 101 needs to train the vocal conversion system, it can obtain multiple reference audio samples to be converted from server 102. These multiple reference audio samples include at least multiple audio samples from the singer in different vocal registers. After obtaining the multiple reference audio samples of the first target timbre from server 102, terminal device 101 can input these multiple reference audio samples into a timbre encoder to obtain the timbre encoding vector of the multiple reference audio samples output by the timbre encoder. The PPG phoneme posterior probabilities and the pitch fundamental frequency of the vocal to be converted are then input into the text encoder of the vocal conversion system to obtain the prior distribution parameters of the vocal content to be converted output by the text encoder. Based on the prior distribution parameters, sampling is performed from the PPG phoneme posterior probabilities and the pitch fundamental frequency of the vocal to be converted to obtain the text sample value vector of the vocal content to be converted. Finally, the text sample value vector of the vocal content to be converted and the timbre encoding vectors of multiple reference audios are input into the timbre perception attention mechanism module to determine a new timbre encoding vector based on the timbre encoding vectors of multiple reference audios and the text sample value vector of the vocal content to be converted. The new timbre encoding vector is then used as a new input to the vocal conversion system to train the vocal conversion system.

[0051] For ease of understanding, the training method of the singing voice conversion system in this application embodiment will be described below. This training method can be applied to a terminal device or server, or to a mini-program installed on a terminal device or server. The training process of the singing voice conversion system will be described in detail below; please refer to [link to relevant documentation]. Figure 2 , Figure 2 This is a schematic diagram of an embodiment of the training method for the singing voice conversion system in this application:

[0052] Specifically, the singing voice conversion system in this application embodiment adds a timbre encoder and a timbre perception attention mechanism module to the original singing voice conversion system. For ease of understanding, Figure 3 A schematic diagram of the singing voice conversion system is provided.

[0053] 201. Obtain multiple reference audios of the first target timbre, wherein the multiple reference audios include at least multiple audios of the singer whose voice is to be converted in different vocal registers;

[0054] Unlike existing technologies that do not fine-tune the original singing voice conversion model, resulting in a low similarity between the timbre of the synthesized audio obtained by the original singing voice conversion system and the target timbre of the singer to be converted, the embodiments of this application add a timbre encoder and a timbre perception attention mechanism module to the original singing voice conversion system, and use multiple reference audios of the first target timbre as conversion sample inputs to the timbre encoder and timbre perception attention mechanism module.

[0055] Specifically, in this embodiment, the multiple reference audios of the first target timbre are multiple audios of the singer whose voice is to be converted in different vocal registers. For example, when the voice to be converted is sung by user A, the multiple reference audios of the first target timbre are multiple audios sung by user A in different vocal registers. Of course, if the voice to be converted is sung by user B, the multiple reference audios of the first target timbre are multiple audios sung by user B in different vocal registers. Here, there is no specific restriction on the singer whose voice is to be converted.

[0056] Furthermore, the multiple reference audios here are songs sung or spoken by the singer whose voice is to be converted. Because the same user uses different resonance methods when singing in different registers (such as high register, middle register, and low register), the timbre is also different. In order to select the best timbre that matches the content of the song to be converted, the timbre of the multiple reference audios generally needs to be as rich as possible. Therefore, the multiple reference audios in this application include at least multiple audios of the singer whose voice is to be converted in different registers. For example, when the sample to be converted is user A, the multiple reference audios here include multiple audios of user A in different registers, such as audios of user A in the high register, audios in the middle register, and audios in the low register.

[0057] It should be noted that the high-pitched and low-pitched audio here can be the high-pitched and low-pitched portions of the same song sung by the singer whose voice is to be converted, or the high-pitched and low-pitched portions of different songs sung by the singer whose voice is to be converted. Furthermore, in the embodiments of this application, the high-pitched and low-pitched audio can be the same song as the voice to be converted, or they can be different songs. No specific restrictions are made here.

[0058] 202. Input the multiple reference audios into the timbre encoder to perform timbre encoding on the multiple reference audios, and obtain the timbre encoding vector of the multiple reference audios output by the timbre encoder;

[0059] After obtaining multiple reference audios, the multiple reference audios are input into the timbre encoder to perform timbre encoding on the multiple reference audios, and the timbre encoding vector of the multiple reference audios is output by the timbre encoder.

[0060] Specifically, the timbre encoder in this application embodiment can be a stacked multi-head attention mechanism to simultaneously focus on different parts of the input and capture different levels of induced information through multiple independent attention heads. Each head has its own weight matrix to capture different combinations of input features. The parallel processing capability of this multi-head attention mechanism improves the model's global perception capability and enables it to better understand and process contextual relationships.

[0061] 203. Input the PPG phoneme posterior probability of the singing voice to be converted and the Pitch fundamental frequency of the singing voice to be converted into the text encoder to obtain the prior distribution parameters of the singing voice content to be converted output by the text encoder.

[0062] In this embodiment, the singing-voice-conversion model (SVC) refers to a model that, when given a sample of singing audio to be converted and an audio sample of the target timbre, can retain the singing content in the output audio and convert the singing timbre to the target timbre. In this embodiment, the singing-voice-conversion model can be a FastSpeech model, a VITS model, or a diffusion model, etc. There are no specific restrictions on the singing-voice-conversion model here.

[0063] Furthermore, all singing conversion models include at least a text encoder. Therefore, in this embodiment, the phonetic posterior probability (PPG) and the fundamental frequency (Pitch) of the singing to be converted are input into the text encoder to obtain the prior distribution parameters of the singing content to be converted output by the text encoder. Here, the prior distribution parameters are functions used to predict the timbre and fundamental frequency distribution. When the prior distribution is a normal distribution, the prior distribution parameters can be the mean and variance.

[0064] Furthermore, the text encoder in this application embodiment can be a multi-head attention mechanism module with the same architecture as the timbre encoder.

[0065] 204. Based on the prior distribution parameters, samples are taken from the posterior probability of the PPG phonemes of the song to be converted and the fundamental frequency of the pitch of the song to be converted to obtain the text sample value vector of the song content to be converted.

[0066] Once the prior distribution parameters are obtained, they can be used to sample from the phoneme posterior probability (PPG) and the fundamental frequency (Pitch) of the song to be converted, so as to obtain the text sample value vector of the song content to be converted.

[0067] In this embodiment, the vocal melody to be converted can be multiple segments of the same song sung by the same singer, or multiple segments of different songs sung by the same singer. Furthermore, after obtaining the vocal melody to be converted, this embodiment also needs to extract the phoneme posterior probability (PPG) and the fundamental frequency (Pitch) of the vocal melody to be converted. When proposing the phoneme posterior probability (PPG), the vocal melody to be converted can be input into a Whisper model, a Hubert model, or a Librosa model to obtain the phoneme posterior probability (PPG). When extracting the fundamental frequency (Pitch), the vocal melody to be converted can be input into an Rmvte vocal melody extraction model based on a recurrent neural network to extract the fundamental frequency (Pitch) of the vocal melody to be converted.

[0068] It should be noted that when using the above model to extract the phoneme posterior probability (PPG) and the fundamental frequency (Pitch) of the singing voice to be converted, the extracted phoneme posterior probability (PPG) and the fundamental frequency (Pitch) of the singing voice to be converted are both in units of audio frames. Therefore, in the embodiments of this application, sampling is performed from the PPG phoneme posterior probability and the fundamental frequency (Pitch) of the singing voice to be converted according to the prior distribution parameters, and the resulting text sampling value vector is also frame-level singing content encoding.

[0069] 205. Input the text sample value vector of the singing content to be converted and the timbre encoding vector of multiple reference audios into the timbre perception attention mechanism module to determine a new timbre encoding vector based on the timbre encoding vector of multiple reference audios and the text sample value vector of the singing content to be converted.

[0070] After obtaining the text sample value vector of the singing content to be converted and the timbre encoding vectors of multiple reference audios, the text sample value vector of the singing content to be converted and the timbre encoding vectors of multiple reference audios are input into the timbre perception attention mechanism module. In this embodiment, the timbre perception attention mechanism module is the attention mechanism in the transformer, and the formula for the attention mechanism is:

[0071]

[0072] Wherein, the content encoding vector is referred to as Q, Let K be the dimension of the timbre encoding vector, and let K and V be the timbre encoding vectors respectively.

[0073] The process of determining a new timbre encoding vector based on the timbre encoding vector of multiple reference audio sources and the text sample value vector of the singing content to be converted is described below, using the formula above:

[0074] 1. As described in step 204, the text sample value vector of the singing content to be converted is used to represent the content encoding of the singing, that is, the text sample value vector of the singing content to be converted is Q in the above formula, and the timbre encoding vectors of multiple reference audios are respectively used as K and V in the above formula.

[0075] 2. In the above formula, the attention score is calculated first. The attention score is mainly used to calculate the similarity between vector Q and vector K, and the attention score is as follows:

[0076] in, is the dimension of the timbre encoding vector, used to scale the dot product to prevent the softmax input from being too large, which would result in a very small gradient.

[0077] 3. After obtaining the attention score, calculate the weight of the attention score. Assuming the weight of the attention score is α, then calculate α according to the following formula:

[0078] α = softmax(score(Q,K))

[0079] 4. Weighted Summation: After obtaining the attention weights, the timbre encoding vector is then weighted and summed according to these attention weights to obtain a new timbre encoding vector. The weighted summation formula is as follows:

[0080] Attention(Q,K,V)=αV

[0081] Through the above process, a new timbre encoding vector can be determined based on the text sample value vector of the song content to be converted and the timbre encoding vectors of multiple reference audios.

[0082] In other words, after inputting the text sample value vector of the singing content to be converted and the timbre encoding vectors of multiple reference audios into the timbre perception attention mechanism module, the timbre perception attention mechanism module can determine the new timbre encoding vector based on the text sample value vector of the singing content to be converted and the timbre encoding vectors of multiple reference audios.

[0083] 206. Using the new timbre encoding vector as the new input added to the singing voice conversion system, calculate the reconstruction loss of the singing voice conversion system, and train the singing voice conversion system based on the reconstruction loss and the backpropagation algorithm.

[0084] After obtaining the new timbre encoding vector output by the timbre perception attention mechanism module, the new timbre encoding vector is used as a new input to the singing voice conversion system. The reconstruction loss of the singing voice conversion system is calculated, and the singing voice conversion system is trained based on the reconstruction loss and the backpropagation algorithm.

[0085] Specifically, in calculating the reconstruction loss of the vocal conversion system, this embodiment calculates the loss between the Mel spectrum of the vocal to be converted and the Mel spectrum of the synthesized audio. The horizontal axis of the Mel spectrum represents time (used to represent the singing time of the vocal to be converted), and the vertical axis represents frequency (used to represent the pitch of the vocal to be converted). In order to reduce the loss between the Mel spectrum of the vocal to be converted and the Mel spectrum of the synthesized audio during the training of the vocal conversion system, this embodiment uses the same vocal to be converted and multiple reference audios during the training of the vocal conversion system. Human audio, that is, if the singing voice to be converted is sung by user A, then the multiple reference audios are also multiple vocal range segments sung by user A. And it is easy to understand that the singing voice conversion system after training learns the ability to generate the timbre of the synthesized audio using the timbre of multiple reference audios. For example, in the model inference stage, when the singing voice to be converted is audio sung by user A, and the multiple reference audios are audio sung by the target singer (such as Liu Xihua), then the generated synthesized audio is the content of the singing voice to be converted sung by user A using the timbre of the target singer (Liu Xihua).

[0086] Specifically, since the singing voice conversion system adds a timbre encoder and a timbre perception attention mechanism module to the original singing voice conversion system, and the original singing voice conversion system can be a FastSpeech model, VITS model, or diffusion model, the training method used when converting singing voices using the new timbre encoding vector varies depending on each model. The following describes the training process of the singing voice conversion system when including different original singing voice conversion systems, which will not be elaborated here.

[0087] In this embodiment, a timbre encoder and a timbre perception attention mechanism module are added to the singing system. Multiple reference audios of the first target timbre are input into the timbre encoder. These multiple reference audios contain multiple audios of the singer whose voice is to be converted in different vocal registers. Because the resonance methods used in singing high and low registers are different, the timbre of the singer whose voice is to be converted is also different in the multiple reference audios. This allows the trained singing conversion system to learn the target timbre that is suitable for the singing voice to be converted from the multiple reference audios based on the singing content representation vector of the singing voice to be converted. Then, based on the target timbre that is suitable for the singing voice to be converted, the timbre of the singing voice to be converted is synthesized, thereby improving the similarity between the timbre of the synthesized singing voice and the timbre of the singing voice to be converted.

[0088] The following describes the training process of a singing voice conversion system when different singing voice conversion systems are included:

[0089] 1. If the vocal conversion system includes the VITS vocal conversion system based on a variational autoencoder;

[0090] Specifically, when the vocal conversion system includes the VITS vocal conversion system based on a variational autoencoder, for ease of understanding, Figure 4 A schematic diagram of the vocal conversion system including VITS is provided as an optional embodiment. Figure 4 The VITS also includes: a posteriori encoder, a FLOW streaming module, and a decoder. Figure 2 Based on the examples, the training process of VITS will be described below. Please refer to [link / reference]. Figure 5 :

[0091] 501. Input the linear spectrum of the singing voice to be converted into the posterior encoder to obtain the posterior distribution parameters output by the posterior encoder module;

[0092] Specifically, in this embodiment, in addition to inputting the PPG phoneme posterior probability and the pitch fundamental frequency of the singing voice to be converted into the text encoder, and inputting multiple reference audios of the sample to be converted into the timbre encoder, the linear spectrum of the singing voice to be converted can also be input into the posterior encoder to obtain the posterior distribution parameters output by the posterior encoder module. In this embodiment, the role of the posterior encoder is to make the posterior distribution parameters output by the posterior encoder module as close as possible to the prior distribution parameters output by the text encoder, so that the loss value between the first output of the singing voice after passing through the posterior encoder and the second output after passing through the text encoder is as small as possible.

[0093] In this embodiment, the posterior encoder may include WavNet, a deep convolutional neural network architecture. Unlike traditional recurrent neural networks (RNNs), WavNet uses causal convolution layers, which allows the model to consider only previous input samples during forward propagation, avoiding the problems caused by information loops. This is very effective when processing variable-length audio signals.

[0094] The core idea of ​​WavNet is to process audio sequences using Temporal Convolutional Blocks (TCBs) and Residual Connections. Each TCB contains multiple small convolutional kernels that slide frame-by-frame along the timeline while maintaining a global awareness of the input. Furthermore, due to its sparse connection design, WavNet can capture long-term dependencies and learn more complex local features.

[0095] As an optional embodiment, after obtaining the singing voice to be converted, this embodiment of the application may input the singing voice to be converted into the Librosa library to obtain the linear spectrum of the singing voice to be converted, and further input the linear spectrum of the singing voice to be converted into the a posteriori encoder.

[0096] 502. Based on the posterior distribution parameters, sample from the linear spectrum to obtain a sampled value vector of the linear spectrum;

[0097] After obtaining the posterior distribution parameters output by the posterior encoder in step 501, the posterior distribution parameters are used to sample from the linear spectrum of the song to be converted, thereby obtaining the sampled value vector of the linear spectrum.

[0098] As an optional embodiment, the posterior distribution parameter in the embodiments of this application may also be a distribution parameter that includes the mean and variance.

[0099] 503. Input the sample value vector of the linear spectrum into the FLOW stream module to map the sample value vector of the linear spectrum from the complex distribution of audio to the simple distribution of text, so as to obtain the mapped sample value vector of the linear spectrum;

[0100] After obtaining the sampled values ​​of the linear spectrum, this embodiment further inputs the sampled value vector of the linear spectrum into the FLOW stream module to map the sampled value vector of the linear spectrum from the complex distribution of audio to the simple distribution of text, so as to obtain the mapped sampled value vector of the linear spectrum.

[0101] Specifically, the FLOW module, as a mapping function, maps numerical distributions from one distribution to another. Because it is a mapping function, the FLOW module is reversible. When the FLOW module transforms the first numerical distribution into the second numerical distribution, the inverse FLOW module transforms the second numerical distribution into the first numerical distribution.

[0102] Because the linear spectrum in this embodiment is a type of audio data, while the PPG phoneme posterior probability of the singing voice to be converted and the Pitch fundamental frequency of the singing voice to be converted are types of text data, the audio data has a higher complexity than the text data. In order to ensure that both have the same data distribution when calculating the loss value, the function of the FLOW stream module in this embodiment is to map the sample value vector of the linear spectrum from the complex distribution of audio to the simple distribution of text, so as to obtain the mapped sample value vector of the linear spectrum.

[0103] 504. Based on the prior distribution of the text sample value vector of the song content to be converted and the distribution of the sample value vector of the mapped linear spectrum, calculate the KL divergence between the posterior distribution and the prior distribution to obtain the KL loss.

[0104] After obtaining the sampled value vector of the mapped linear spectrum in step 503, the KL divergence between the prior distribution and the posterior distribution of the sampled value vector of the text of the song content to be converted is calculated based on the prior distribution of the text sampled value vector and the posterior distribution of the sampled value vector of the mapped linear spectrum, so as to obtain the KL loss of the model.

[0105] Specifically, KL divergence (Kullback-Leibler Divergence), also known as information gain or relative entropy, is a statistic that measures the difference between two probability distributions. If distribution P is exactly equal to distribution Q, the KL divergence is 0; while the greater the difference between distribution P and distribution Q, the larger the KL divergence.

[0106] 505. Concatenate or add the sampled value vector of the linear spectrum and the new phoneme vector to obtain the fused vocal vector;

[0107] After obtaining the sampled value vector of the linear spectrum and the new phoneme vector through the above steps, the sampled value vector of the linear spectrum and the new phoneme vector are further fused to obtain the fused singing vector.

[0108] Specifically, when fusing the sampled value vector of the linear spectrum and the new phoneme vector, the fusion operation can be either concatenation or addition; there are no restrictions on the specific means of fusion operation here.

[0109] 506. The fused vocal vector is input to the decoder to obtain the synthesized vocal audio output by the decoder, wherein the decoder includes an upsampling module;

[0110] After obtaining the fused vocal vector, the fused vocal vector is input into the decoder to obtain the synthesized vocal audio output by the decoder.

[0111] As an optional embodiment, the decoder in this application includes an upsampling module to decode the fused vocal vectors to obtain the synthesized vocal audio.

[0112] 507. Calculate the reconstruction loss based on the Mel spectrum of the singing voice to be converted and the Mel spectrum of the synthesized singing voice audio;

[0113] After the decoder outputs the synthesized vocal audio, in order to facilitate the calculation of reconstruction loss, this embodiment of the application further extracts the Mel spectrum features of the synthesized vocal audio and the Mel spectrum features of the vocal audio to be converted.

[0114] As an optional embodiment, when extracting the Mel spectrum features of singing, the following steps can be performed: first, the time-domain signal of the singing is Fourier transformed to the frequency domain to obtain the frequency domain features of the singing; then, the frequency domain signal of the singing is processed using a filter bank with Mel frequency scale to obtain the Mel spectrum.

[0115] Furthermore, after obtaining the Mel spectrum features of the synthesized singing audio and the Mel spectrum features of the singing to be converted, the loss between the Mel spectrum features of the synthesized singing audio and the Mel spectrum features of the singing to be converted is calculated using a preset loss function. The loss function here can be the mean squared error loss function, the cross-entropy loss function, or the logarithmic loss function, etc. There are no specific restrictions on the loss function here.

[0116] 508. The VITS vocal conversion system based on variational autoencoder is trained using reconstruction loss, KL loss and backpropagation algorithm.

[0117] After obtaining the reconstruction loss in step 507, the VITS vocal conversion system based on variational autoencoder is further trained using the reconstruction loss, KL loss and backpropagation algorithm to obtain the trained vocal conversion system.

[0118] In this embodiment of the application, when the singing conversion system includes VITS, the training process of the VITS system is described in detail. The VITS system uses the FLOW module, thereby improving the accuracy of calculating KL loss. Furthermore, this embodiment of the application uses reconstruction loss and KL loss to train the VITS system, which also improves the timbre similarity between the synthesized singing audio output by the VITS system and the input singing audio.

[0119] based on Figure 5 In the embodiments described above, to further improve the accuracy of the synthesized vocal audio output by the VITS system, a discriminator can be added after the decoder. For ease of understanding, Figure 6 Another architectural diagram of the VITS system is given. Figure 2 Based on the implementation examples, Figure 7 This is another embodiment of the VITS system training process described in this application:

[0120] 701. Input the linear spectrum of the singing voice to be converted into the posterior encoder to obtain the posterior distribution parameters output by the posterior encoder module;

[0121] 702. Based on the posterior distribution parameters, sample from the linear spectrum to obtain a sampled value vector of the linear spectrum;

[0122] 703. Input the sample value vector of the linear spectrum into the FLOW stream module to map the sample value vector of the linear spectrum from the complex distribution of audio to the simple distribution of text, so as to obtain the mapped sample value vector of the linear spectrum.

[0123] 704. Based on the prior distribution of the text sample value vector of the song content to be converted and the distribution of the sample value vector of the mapped linear spectrum, calculate the KL divergence between the posterior and prior distributions to obtain the KL loss.

[0124] 705. Concatenate or add the sampled value vector of the linear spectrum and the new phoneme vector to obtain the fused vocal vector;

[0125] 706. The fused vocal vector is input to the decoder to obtain the synthesized vocal audio output by the upsampling module, wherein the decoder includes the upsampling module;

[0126] 707. Calculate the reconstruction loss based on the Mel spectrum features of the singing voice to be converted and the Mel spectrum features of the synthesized singing voice audio.

[0127] It should be noted that steps 701 to 707 in the embodiments of this application are different from those in the present application. Figure 5Steps 501 to 507 in the embodiments are similar and will not be repeated here.

[0128] 708. Input the synthesized singing audio into the discriminator to obtain the predicted discrimination result output by the discriminator;

[0129] The improved VITS system with the addition of a discriminator is equivalent to a GAN adversarial neural network. In order to improve the accuracy of the synthesized singing audio output by the decoder, the embodiments of this application further add a discriminator at the back end of the decoder and input the synthesized singing audio output by the decoder into the discriminator, so that the discriminator can discriminate the synthesized singing audio output by the decoder to obtain the predicted discrimination result output by the discriminator.

[0130] Specifically, without a discriminator, that is... Figure 5 The improved VITS system in the middle is equivalent to a generator, while Figure 7 The middle is in Figure 5 A discriminator is added after the generator to discriminate the synthesized singing audio generated by the generator in order to obtain the predicted discrimination result output by the discriminator.

[0131] Specifically, the prediction and identification result in this application embodiment can be a binary classification result. For example, the prediction and identification result can be whether the synthesized singing audio is the audio of a certain singer. If yes, output 1; if no, output 0.

[0132] 709. Calculate the adversarial loss based on the actual identification result and the predicted identification result of the synthesized singing audio;

[0133] After obtaining the prediction results output by the discriminator, the predicted discrimination results and the real discrimination results are further used to calculate the adversarial loss, and the adversarial loss is used to train the improved VITS system.

[0134] 710. The improved VITS vocal conversion system based on variational autoencoder is trained according to the reconstruction loss, the KL loss and the adversarial loss.

[0135] Since the improved VITS system in this application embodiment introduces reconstruction loss, KL loss and adversarial loss respectively, after obtaining the above losses, the improved VITS system can be updated in reverse using the above losses and backpropagation algorithm to continuously update the model parameters in the improved VITS system until the improved VITS system converges.

[0136] In this embodiment, a discriminator is further introduced into the improved VITS system, and the adversarial loss of the discriminator is used to update the model of the improved VITS system, thereby further improving the accuracy of the synthesized singing audio output by the improved VITS system.

[0137] 2. If the singing conversion system includes the Fastspeeh fast speech synthesis model

[0138] Specifically, if the singing conversion system includes the Fastspeeh fast speech synthesis model, then the Fastspeeh fast speech synthesis model further includes a text encoder and a decoder. Both the text encoder and decoder employ stacked multi-head attention mechanism modules. For ease of description, it is assumed that the text encoder includes a first multi-head attention mechanism module, and the decoder includes a second multi-head attention mechanism module. Figure 8 A schematic diagram of the Fastspeeh fast speech synthesis model is given. Figure 2 Based on the examples, the training process of the Fastspeeh fast speech synthesis model is described in detail below. Please refer to [link / reference]. Figure 9 :

[0139] 901. The text sample value vector of the singing content to be converted and the new timbre encoding vector are concatenated or added to obtain the fused singing vector. The fused singing vector is then input into the decoder to obtain the Mel spectrum of the synthesized singing audio output by the decoder.

[0140] exist Figure 2 Based on the embodiments, after obtaining the new timbre encoding vector output by the timbre perception attention mechanism module and the text sampling value vector of the singing content to be converted, the embodiments of this application concatenate or add the text sampling value vector of the singing content to be converted and the new timbre encoding vector to obtain the fused singing vector. The fused singing vector is then input to the decoder, that is, input to the second multi-head attention mechanism module, to obtain the Mel spectrum of the synthesized singing audio output by the second multi-head attention mechanism module.

[0141] It is important to note that due to the different model architectures, when the vocal conversion system includes the VITS system, the decoder output is the synthesized vocal audio, while when the vocal conversion system includes Fastspeeh, the decoder output is the Mel spectrum of the synthesized vocal audio.

[0142] 902. Calculate the reconstruction loss based on the Mel spectrum of the synthesized vocal audio, the Mel spectrum of the vocal audio to be converted, and the preset loss function;

[0143] After obtaining the Mel spectrum of the synthesized vocal audio output by the decoder, the reconstruction loss is further calculated based on the Mel spectrum of the synthesized vocal audio, the Mel spectrum of the vocal audio to be converted, and the preset loss function.

[0144] The process of extracting Mel spectrum features and the preset loss function are similar to those described in the above embodiments, and will not be repeated here.

[0145] 903. Train the Fastspeeh fast speech synthesis model based on the reconstruction loss and backpropagation algorithm.

[0146] When the singing conversion system includes the Fastspeeh model, the loss only includes the reconstruction loss between the Mel spectrum features of the synthesized singing audio and the Mel spectrum features of the singing to be converted, so that the Fastspeeh system can be trained quickly based on the reconstruction loss.

[0147] Because in this embodiment, when the singing conversion model includes Fastspeeh, the loss function only includes reconstruction loss, and the Fastspeeh model structure is simpler than the VITS model structure, the configuration requirements for terminal processing are reduced, and the training process of the improved Fastspeeh model is improved.

[0148] based on Figure 9 In the aforementioned embodiment, to further improve the accuracy of the Mel spectrum of the synthesized singing audio output by the Fastspeeh model, it is also possible to... Figure 9 A discriminator is added to the back end of the decoder. For ease of understanding, Figure 10 A schematic diagram of the architecture of the Fastspeeh fast speech synthesis model is given. Figure 11 Another example of the training process for the Fastspeeh fast speech synthesis model is given:

[0149] 1101. The text sample value vector of the singing content to be converted and the new timbre encoding vector are concatenated or added to obtain the fused singing vector. The fused singing vector is then input into the decoder to obtain the Mel spectrum of the synthesized singing audio output by the decoder.

[0150] 1102. Calculate the reconstruction loss based on the Mel spectrum of the synthesized vocal audio, the Mel spectrum of the vocal audio to be converted, and the preset loss function;

[0151] Steps 1101 to 1102 are similar to steps 901 to 902, and will not be repeated here.

[0152] 1103. Input the Mel spectrum of the synthesized singing audio into the discriminator to obtain the predicted discrimination result output by the discriminator;

[0153] 1104. Calculate the adversarial loss based on the actual identification result of the Mel spectrum of the synthesized singing audio and the predicted identification result;

[0154] The descriptions of steps 1103 to 1104 are similar to those of steps 708 to 709, and will not be repeated here.

[0155] 1105. Train the Fastspeeh fast speech synthesis model based on reconstruction loss, adversarial loss and backpropagation algorithm.

[0156] Since the reconstruction loss is obtained in step 1102 and the adversarial loss is obtained in step 1104, the reconstruction loss and adversarial loss, along with the backpropagation algorithm, are used to train the model parameters of the Fastspeeh fast speech synthesis model until the model converges.

[0157] The embodiments of this application are as follows: Figure 9 Based on the previous implementation, a discriminator and adversarial loss were further introduced to train the Fastspeeh speech synthesis model, thereby improving the accuracy of the Mel spectrum features of the synthesized singing audio output by the Fastspeeh speech synthesis model.

[0158] III. If the singing conversion system includes a Diffusion-based generation model

[0159] Specifically, if the singing conversion system includes a Diffusion-based generation model, then the Diffusion-based generation model further includes a text encoder and a decoder. The text encoder includes a third multi-head attention mechanism module, and the decoder includes a U-net module or a DiT module. The schematic diagram of the Diffusion-based generation model architecture is shown below. Figure 8 The architecture diagram of the Fastspeeh fast speech synthesis model is similar, except that the modules in the decoder are slightly different. The training process of the Diffusion diffusion generation model will be described below; please refer to [link / reference]. Figure 12 :

[0160] 1201. The text sample value vector of the singing content to be converted and the new timbre encoding vector are concatenated or added to obtain the fused singing vector. The fused singing vector is input to the decoder to obtain the Mel spectrum of the synthesized singing audio output by the decoder.

[0161] Because the singing conversion system in this embodiment includes the Diffusion diffusion generation model, and unlike the Fastspeeh fast speech synthesis model, the decoder in this embodiment uses the U-net module or the DiT module. Since the U-net module uses a residual training network, when the decoder uses the U-net model, the convergence speed of the model can be accelerated and the training time of the Diffusion diffusion generation model can be shortened.

[0162] 1202. Calculate the reconstruction loss based on the Mel spectrum features of the synthesized singing audio, the Mel spectrum features of the singing to be converted, and the preset loss function;

[0163] 1203. The Diffusion-based generative model is trained according to the reconstruction loss and backpropagation algorithm.

[0164] The process from step 1202 to step 1203 is similar to the description of steps 902 to 903, and will not be repeated here.

[0165] In this application embodiment, when the singing conversion system includes the Diffusion diffusion generation model, the training process of the Diffusion diffusion generation model is described in detail. Furthermore, when the decoder in the Diffusion diffusion generation model is the U-net module, the convergence speed of the model can be accelerated, and the training time of the Diffusion diffusion generation model can be shortened.

[0166] based on Figure 12 In the embodiments described above, to improve the accuracy of the synthesized vocal audio output by the Diffusion diffusion generation model, this application embodiment can further add a discriminator after the decoder in the Diffusion diffusion generation model. Another schematic diagram of the Diffusion diffusion generation model is shown below. Figure 10 The architecture diagram of the FastSpeeh fast speech synthesis model is similar. Figure 13 Another example of the training process for a diffusion-based generative model is given:

[0167] 1301. The text sample value vector of the singing content to be converted and the new timbre encoding vector are concatenated or added to obtain the fused singing vector. The fused singing vector is input to the decoder to obtain the Mel spectrum of the synthesized singing audio output by the decoder.

[0168] 1302. Calculate the reconstruction loss based on the Mel spectrum features of the synthesized singing audio, the Mel spectrum features of the singing to be converted, and the preset loss function;

[0169] The descriptions of steps 1301 to 1302 are similar to those of steps 901 to 902, and will not be repeated here.

[0170] 1303. Input the Mel spectrum of the synthesized singing audio into the discriminator to obtain the predicted discrimination result output by the discriminator;

[0171] 1304. Calculate the adversarial loss based on the actual identification result of the Mel spectrum of the synthesized singing audio and the predicted identification result;

[0172] 1305. The Diffusion-based generative model is trained based on the reconstruction loss, the adversarial loss, and the backpropagation algorithm.

[0173] The descriptions of steps 1303 to 1305 are similar to those of steps 1103 to 1105, and will not be repeated here.

[0174] The embodiments of this application are as follows: Figure 12 Based on the previous implementation, a discriminator and adversarial loss were further introduced to train the Diffusion diffusion generation model, thereby improving the accuracy of the Mel spectrum features of the synthesized singing audio output by the Diffusion diffusion generation model.

[0175] The training method of the singing voice conversion system in the embodiments of this application has been described in detail above. The method for generating audio based on the singing voice conversion system in the embodiments of this application will be described below. Please refer to [link to relevant documentation]. Figure 14 :

[0176] Specifically, in this embodiment, the singing conversion system includes at least a text encoder, a timbre encoder, a timbre-aware attention mechanism module, and a decoder. For ease of understanding, Figure 15 A schematic diagram of the singing voice conversion architecture is provided, and the methods for generating audio based on the singing voice conversion system include:

[0177] 1401. Obtain multiple reference audios of the second target timbre, wherein the multiple reference audios include at least multiple audios of different singers in different vocal registers;

[0178] In this embodiment of the application, the multiple reference audios for the second target timbre include at least multiple audios from different singers in different vocal registers. Specifically, when generating the target timbre audio using the vocal conversion system, the vocal sample to be converted can be converted into the timbre of different singers, such as converting the vocal sample to be converted into the timbre of Liu Xihua or Zhou Y, that is, converting the vocal sample to be converted into the timbre of other singers. Therefore, in this embodiment of the application, the multiple reference audios include at least multiple audios from different singers in different vocal registers. In addition, because each singer uses different resonance methods when singing in different vocal registers, that is, different resonance methods are used when singing in the high and low registers, in order to convert the vocal sample to be converted into the most suitable target timbre, multiple audios from different singers in different vocal registers (such as the high, middle and low registers) are needed so that the best target timbre suitable for the vocal sample to be converted can be found during the conversion process.

[0179] Furthermore, multiple audio recordings from different singers in different vocal registers can be audio recordings of different singers performing the same song, or audio recordings of different singers performing different songs. There are no specific restrictions on the content of these multiple audio recordings from different singers in different vocal registers.

[0180] 1402. Input multiple reference audios into the timbre encoder to perform timbre encoding on the multiple reference audios, and obtain the timbre encoding vector of the multiple reference audios output by the timbre encoder;

[0181] After obtaining multiple reference audios from different singers in different vocal registers, these multiple reference audios are input into the timbre encoder to perform timbre encoding on the multiple reference audios of the singer's voice, resulting in the timbre encoding vector of the multiple reference audios output by the timbre encoder.

[0182] Specifically, the timbre encoder in this application embodiment can be a stacked multi-head attention mechanism. This is because the multi-head attention mechanism can simultaneously focus on different parts of the input and capture different levels of induced information through multiple independent attention heads. Each head has its own weight matrix to capture different combinations of input features. The parallel processing capability of this multi-head attention mechanism improves the model's global perception capability and enables it to better understand and process contextual relationships.

[0183] 1403. Input the posterior probability of the PPG phonemes of the song to be converted and the fundamental frequency of the pitch of the song to be converted into the text encoder to obtain the prior distribution parameters of the song content to be converted output by the text encoder.

[0184] Specifically, in this embodiment, the PPG phoneme posterior probability and the pitch fundamental frequency of the singing voice to be converted are both used to characterize the singing content of the singer. In this embodiment, after obtaining the singing voice of the sample to be converted, the singing voice to be converted can be input into a Whisper model, a Hubert model, or a Librosa model to obtain the phoneme posterior probability (PPG) of the singing voice to be converted, and the singing voice to be converted can be input into the Rmvte singing melody extraction model based on a recurrent neural network to extract the pitch fundamental frequency of the singing voice to be converted.

[0185] In this embodiment of the application, the text encoder is a multi-head attention mechanism module with the same architecture as the timbre encoder. When the phoneme posterior probability (PPG) and the fundamental frequency (Pitch) of the singer's voice are input into the text encoder, the prior distribution parameters of the predicted phoneme distribution and fundamental frequency distribution in the singer's voice can be obtained from the text encoder output. When the prior distribution is a normal distribution, the prior distribution parameters can be the mean and variance.

[0186] Furthermore, in this embodiment, the sample to be converted is a user whose singing voice needs to be converted. For example, if user A's singing voice needs to be converted into the target timbre, then the sample to be converted is user A. If user B's singing voice needs to be converted into the target timbre, then the user to be converted is user B. Therefore, in this embodiment, the sample to be converted is a different user whose timbre needs to be converted.

[0187] 1404. Based on the prior distribution parameters, samples are taken from the posterior probability of the PPG phonemes of the song to be converted and the fundamental frequency of the pitch of the song to be converted to obtain the text sample value vector of the song content to be converted.

[0188] After obtaining the prior distribution parameters, the prior distribution parameters are used to sample from the phoneme posterior probability (PPG) and the fundamental frequency (Pitch) of the song to be converted, so as to obtain the text sample value vector of the song content to be converted.

[0189] It should be noted that, because the phoneme posterior probability (PPG) of the vocal melody to be converted extracted using the Whisper model, Hubert model, or Librosa model in this embodiment is at the frame level, and the fundamental frequency (Pitch) of the vocal melody to be converted extracted by the Rmvte recurrent neural network-based vocal melody extraction model is also at the frame level, the text sampling value vector of the vocal melody to be converted obtained in this embodiment is also a frame-level vocal melody content encoding.

[0190] 1405. Input the text sample value vector of the singing content to be converted and the timbre encoding vector of multiple reference audios into the timbre perception attention mechanism module, and determine the new timbre encoding vector based on the timbre encoding vector of multiple reference audios and the text sample value vector of the singing content to be converted.

[0191] Because a timbre perception attention mechanism module is added in this embodiment, after inputting the text sampling value vector of the singer's vocal content and the timbre encoding vector of multiple reference audios into the timbre perception attention mechanism module, a new timbre encoding vector that matches the text sampling value vector of the singer's vocal content can be obtained from the output of the timbre perception attention mechanism module.

[0192] Because the timbre perception attention mechanism module in this embodiment is the attention mechanism in the transformer, the formula of the attention mechanism is:

[0193]

[0194] In this context, the text sample value vector of the singing content to be converted is used as Q, and the timbre encoding is used as K and V. Let K be the dimension of the timbre encoding vector.

[0195] After the above calculations, a new timbre encoding vector that matches the content of the song to be converted can be obtained.

[0196] 1406. The text sample value vector of the singing content to be converted and the new timbre encoding vector are used as inputs to the decoder to obtain the synthesized singing audio.

[0197] After obtaining the text sample value vector and the new timbre encoding vector of the singing content to be converted, inputting the text sample value vector and the new timbre encoding vector of the singing content to be converted into the decoder will yield the synthesized singing audio output by the decoder.

[0198] Because the new timbre encoding vector in this embodiment is a new timbre encoding vector that matches the text sample value vector of the singing content, determined based on the timbre encoding vectors of multiple reference audios and the text sample value vector of the singing content to be converted, the timbre vector of the synthesized singing audio output by the decoder has a higher timbre similarity to the singer.

[0199] The following is a detailed description of the singing voice conversion system, including different singing voice conversion systems:

[0200] I. If the vocal conversion system includes the VITS vocal conversion system based on a variational autoencoder.

[0201] Specifically, when the singer conversion system includes the VITS vocal conversion system based on a variational autoencoder, then VITS also includes an inverse FLOW streaming module. Figure 16 A schematic diagram of the vocal conversion system including VITS is provided below. The following is a schematic diagram of the process by which VITS outputs the synthesized vocal audio. Please refer to [link / reference]. Figure 17 :

[0202] 1701. Obtain multiple reference audios of the second target timbre, wherein the multiple reference audios include at least multiple audios of different singers in different vocal registers;

[0203] 1702. Input multiple reference audios into the timbre encoder to perform timbre encoding on the multiple reference audios, and obtain the timbre encoding vector of the multiple reference audios output by the timbre encoder;

[0204] 1703. Input the posterior probability of the PPG phonemes of the song to be converted and the fundamental frequency of the pitch of the song to be converted into the text encoder to obtain the prior distribution parameters of the content of the song to be converted output by the text encoder.

[0205] 1704. Based on the prior distribution parameters, samples are taken from the posterior probability of the PPG phonemes of the song to be converted and the fundamental frequency of the pitch of the song to be converted to obtain the text sample value vector of the song content to be converted.

[0206] 1705. Input the text sample value vector of the singing content to be converted and the timbre encoding vector of multiple reference audios into the timbre perception attention mechanism module to determine a new timbre encoding vector based on the timbre encoding vector of multiple reference audios and the text sample value vector of the singing content to be converted.

[0207] It should be noted that steps 1701 to 1705 are similar to steps 1401 to 1405 in description, and will not be repeated here.

[0208] 1706. Input the text sample value vector of the singing content to be converted into the inverse FLOW stream module to map the simple distribution of the text sample value vector of the singing content to be converted to the complex distribution of the audio, so as to obtain the mapped text sample value vector of the singing content to be converted.

[0209] Because when the vocal conversion system includes VITS, a FLOW stream module is introduced during the training process. Therefore, during the derivation phase, VITS also includes an inverse FLOW stream module to map the simple distribution of the text sample value vector of the vocal content to be converted to the complex distribution of the audio, so as to obtain the mapped text sample value vector of the vocal content to be converted.

[0210] It should be noted that the inverse FLOW stream module in this embodiment is the inverse operation of the FLOW stream module in the training process. When the FLOW stream module is the first function, the inverse FLOW stream module is the inverse function of the first function. The function of the FLOW stream module is to convert the complex distribution of audio into a simple distribution of text sample value vector, while the function of the inverse FLOW stream module is to convert the simple distribution of text sample value vector into a complex distribution of audio, so as to obtain the mapped text sample value vector of the singer's singing content.

[0211] 1707. Concatenate or add the mapped text sample value vector of the singing content to be converted and the new timbre encoding vector to obtain the fused singing vector;

[0212] After obtaining the mapped text sample value vector of the singing content to be converted and the new timbre encoding vector, the mapped text sample value vector of the singing content to be converted and the new timbre encoding vector are further fused to obtain the fused singing vector. Here, the fusion operation can be vector concatenation or addition.

[0213] 1708. Input the fused vocal vector into the decoder to obtain the synthesized vocal audio.

[0214] After obtaining the fused vocal vector, the fused vocal vector is input into the decoder to obtain the synthesized vocal audio.

[0215] Specifically, in this embodiment, the decoder is an upsampling module.

[0216] In this embodiment of the application, when the singing conversion system includes VITS based on variational autoencoder, the process of generating synthesized audio is described in detail. In this embodiment of the application, VITS based on variational autoencoder can select new timbre encoding vectors that match the singing content from multiple reference audios of different singers in different vocal ranges. Therefore, the synthesized singing audio has a high timbre similarity to the singer's singing.

[0217] II. If the singing conversion system includes the FastSpeeh fast speech synthesis model or the Diffusion diffusion generation model.

[0218] Specifically, if the singing conversion system includes the Fastspeeh fast speech synthesis model or the Diffusion diffusion generation model, then the architectural diagram of the Fastspeeh fast speech synthesis model or the Diffusion diffusion generation model is as follows: Figure 15 As shown, the process of generating synthesized audio using the FastSpeeh fast speech synthesis model or the Diffusion diffusion generation model is as follows: Figure 18 As shown:

[0219] 1801. Obtain multiple reference audios of the second target timbre, wherein the multiple reference audios include at least multiple audios of different singers in different vocal registers;

[0220] 1802. Input multiple reference audios into the timbre encoder to perform timbre encoding on the multiple reference audios, and obtain the timbre encoding vector of the multiple reference audios output by the timbre encoder;

[0221] 1803. Input the posterior probability of the PPG phonemes of the song to be converted and the fundamental frequency of the pitch of the song to be converted into the text encoder to obtain the prior distribution parameters of the song content to be converted output by the text encoder.

[0222] 1804. Based on the prior distribution parameters, samples are taken from the posterior probability of the PPG phonemes of the song to be converted and the fundamental frequency of the pitch of the song to be converted to obtain the text sample value vector of the song content to be converted.

[0223] 1805. Input the text sample value vector of the singing content to be converted and the timbre encoding vector of multiple reference audios into the timbre perception attention mechanism module to determine a new timbre encoding vector based on the timbre encoding vector of multiple reference audios and the text sample value vector of the singing content to be converted.

[0224] It should be noted that steps 1801 to 1805 are similar to steps 1401 to 1405 in description, and will not be repeated here.

[0225] Furthermore, when the singing voice is converted into the Fastspeeh fast speech synthesis model or the Diffusion diffusion generation model, the text encoder in both models is a multi-head attention mechanism module.

[0226] 1806. The text sample value vector of the singing content to be converted and the new timbre encoding vector are concatenated or added together to obtain the fused singing vector;

[0227] When the singing conversion system includes the Fastspeeh fast speech synthesis model or the Diffusion diffusion generation model, if the text sample value vector of the singing content to be converted and the new timbre encoding vector are obtained, the text sample value vector of the singer's singing content and the new timbre encoding vector are fused to obtain the fused singing vector. The fusion operation here can be vector addition or concatenation.

[0228] 1807. Input the fused vocal vector into the decoder to obtain the synthesized vocal audio.

[0229] After obtaining the fused vocal vector, the fused vocal vector is input into the decoder to obtain the synthesized vocal audio.

[0230] It should be noted that if the singing conversion system is a Fastspeeh fast speech synthesis model, then the decoder is also a multi-head attention mechanism module; while if the singing conversion system is a Diffusion diffusion generation model, then the decoder is a U-net module or a DiT module.

[0231] In this embodiment of the application, when the singing conversion system includes the Fastspeeh fast speech synthesis model or the Diffusion diffusion generation model, the process of synthesizing audio is described in detail. In this embodiment of the application, the Fastspeeh fast speech synthesis model or the Diffusion diffusion generation model can generate a new timbre encoding vector that matches the singing content based on multiple reference audios of different singers in different vocal ranges. Therefore, the synthesized singing audio has a high timbre similarity to the singer's singing.

[0232] Furthermore, the Fastspeeh fast speech synthesis model or Diffusion diffusion generation model in the embodiments of this application has a simpler model structure and reduces the configuration requirements of the terminal processor compared to the VITS singing conversion system based on variational autoencoders.

[0233] This application also provides a computer program product on which a computer program is stored. When the computer program is executed by a processor, it is used to implement the method steps in the above method embodiments.

[0234] This application also provides a computer device, which includes:

[0235] Processor and memory;

[0236] The memory is used to store computer programs, and when the processor executes the computer programs stored in the memory, it is used to implement the method steps in the above method embodiments.

[0237] The computer device may be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device may include, but is not limited to, a processor and memory. Those skilled in the art will understand that the processor and memory are merely examples of a computer device and do not constitute a limitation on the computer device. It may include more or fewer components, or a combination of certain components, or different components. For example, the computer device may also include input / output devices, network access devices, buses, etc.

[0238] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the computer device, connecting various parts of the computer device via various interfaces and lines.

[0239] The memory can be used to store the computer programs and / or modules. The processor implements various functions of the computer device by running or executing the computer programs and / or modules stored in the memory and by calling data stored in the memory. The memory may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function, etc.; the data storage area may store data created according to the use of the terminal, etc. In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, RAM, plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0240] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, is used to implement the method steps in the above method embodiments.

[0241] It is understood that if the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a corresponding computer-readable storage medium. Based on this understanding, all or part of the processes in the above-described embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the above-described method embodiments. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0242] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A training method for a singing voice conversion system, characterized in that, The singing voice conversion system includes at least a text encoder, a timbre encoder, and a timbre-aware attention mechanism module, and the method includes: Obtain multiple reference audios of the first target timbre, wherein the multiple reference audios of the first target timbre include at least multiple audios of the singer whose voice is to be converted in different vocal registers; Multiple reference audios of the first target timbre are input to the timbre encoder to perform timbre encoding on the multiple reference audios of the first target timbre, thereby obtaining the timbre encoding vector of the multiple reference audios of the first target timbre output by the timbre encoder; The PPG phoneme posterior probability and the pitch fundamental frequency of the singing voice to be converted are input into the text encoder to obtain the prior distribution parameters of the singing voice content to be converted output by the text encoder. Based on the prior distribution parameters, samples are taken from the posterior probability of the PPG phonemes of the vocal to be converted and the fundamental frequency of the pitch of the vocal to be converted to obtain the text sample value vector of the vocal content to be converted. The text sampling value vector of the singing content to be converted and the timbre encoding vector of multiple reference audios of the first target timbre are input into the timbre perception attention mechanism module to determine a new timbre encoding vector based on the timbre encoding vector of multiple reference audios of the first target timbre and the text sampling value vector of the singing content to be converted. The new timbre encoding vector is used as a new input to the singing voice conversion system, and the reconstruction loss of the singing voice conversion system is calculated. The singing conversion system is trained based on the reconstruction loss and backpropagation algorithm.

2. The method according to claim 1, characterized in that, If the singing conversion system includes the VITS singing conversion system based on variational autoencoder, the VITS singing conversion system based on variational autoencoder further includes: a posteriori encoder and a FLOW stream module, wherein the text encoder includes a multi-head attention mechanism module; The method further includes: The linear spectrum of the singing voice to be converted is input into the posterior encoder to obtain the posterior distribution parameters output by the posterior encoder module; Based on the posterior distribution parameters, samples are taken from the linear spectrum to obtain a sampled value vector of the linear spectrum; The sample value vector of the linear spectrum is input to the FLOW stream module to map the sample value vector of the linear spectrum from the complex distribution of audio to the simple distribution of text, so as to obtain the mapped sample value vector of the linear spectrum. Based on the prior distribution of the text sample value vector of the song content to be converted and the posterior distribution of the sample value vector of the mapped linear spectrum, the KL divergence between the posterior and prior distributions is calculated to obtain the KL loss.

3. The method according to claim 2, characterized in that, The VITS vocal conversion system based on variational autoencoder also includes a decoder, which includes an upsampling module; The step of using the new timbre encoding vector as a new input to the singing voice conversion system and calculating the reconstruction loss of the singing voice conversion system includes: The sampled value vector of the linear spectrum and the new timbre encoding vector are concatenated or added to obtain the fused singing vector; The fused vocal vector is input into the upsampling module to obtain the synthesized vocal audio output by the upsampling module; The reconstruction loss is calculated based on the Mel spectrum of the singing voice to be converted and the Mel spectrum of the synthesized singing voice audio. The training of the singing conversion system based on the reconstruction loss and backpropagation algorithm includes: The VITS vocal conversion system based on variational autoencoder is trained using the reconstruction loss, the KL loss, and the backpropagation algorithm.

4. The method according to claim 3, characterized in that, The VITS vocal conversion system based on variational autoencoder also includes: a discriminator; The method further includes: The synthesized singing audio is input into the discriminator to obtain the predicted discrimination result output by the discriminator; The adversarial loss is calculated based on the actual identification result and the predicted identification result of the synthesized singing audio. The VITS vocal conversion system based on variational autoencoder is trained using the reconstruction loss, the KL loss, and the backpropagation algorithm, including: The VITS vocal conversion system based on variational autoencoder is trained using the reconstruction loss, the KL loss, the adversarial loss, and the backpropagation algorithm.

5. The method according to claim 2, characterized in that, Before inputting the PPG phoneme posterior probability and the pitch fundamental frequency of the vocal verse to be converted into the text encoder, the method further includes: The singing voice to be converted is input into a first preset model to extract the PPG phoneme posterior probability of the singing voice to be converted. The first preset model includes a Whisper model, a Hubert model, or a Librosa model. The vocal melody to be converted is input into the Rmvte vocal melody extraction model based on a recurrent neural network to extract the pitch fundamental frequency of the vocal melody to be converted. Before inputting the linear spectrum of the singing voice to be converted into the a posteriori encoder, the method further includes: The vocal sample to be converted is input into the Librosa library to obtain the linear spectrum of the vocal sample.

6. The method according to claim 1, characterized in that, If the singing conversion system includes the Fastspeeh fast speech synthesis model, then the Fastspeeh fast speech synthesis model further includes a text encoder and a decoder, wherein the text encoder includes a first multi-head attention mechanism module, and the decoder includes a second multi-head attention mechanism module; Using the new timbre encoding vector as a new input to the singing voice conversion system, the reconstruction loss of the singing voice conversion system is calculated, including: The text sample value vector of the singing content to be converted and the new timbre encoding vector are concatenated or added to obtain the fused singing vector; The fused vocal vector is input into the decoder to obtain the Mel spectrum of the synthesized vocal audio output by the decoder; The reconstruction loss is calculated based on the Mel spectrum of the synthesized vocal audio, the Mel spectrum of the vocal audio to be converted, and a preset loss function.

7. The method according to claim 6, characterized in that, The Fastspeeh fast speech synthesis model also includes: a discriminator; The method further includes: The Mel spectrum of the synthesized singing audio is input into the discriminator to obtain the predicted discrimination result output by the discriminator; The adversarial loss is calculated based on the actual identification result of the Mel spectrum of the synthesized singing audio and the predicted identification result. The Fastspeeh fast speech synthesis model is trained based on the reconstruction loss and backpropagation algorithm, including: The Fastspeeh fast speech synthesis model is trained based on the reconstruction loss, the adversarial loss, and the backpropagation algorithm.

8. The method according to claim 1, characterized in that, If the singing conversion system includes a Diffusion-based generation model, then the Diffusion-based generation model further includes a text encoder and a decoder, wherein the text encoder includes a third multi-head attention mechanism module, and the decoder includes a U-net module or a DiT module; Using the new timbre encoding vector as a new input to the singing voice conversion system, the reconstruction loss of the singing voice conversion system is calculated, including: The text sample value vector of the singing content to be converted and the new timbre encoding vector are concatenated or added to obtain the fused singing vector; The fused vocal vector is input into the decoder to obtain the Mel spectrum of the synthesized vocal audio output by the decoder; The reconstruction loss is calculated based on the Mel spectrum of the synthesized vocal audio, the Mel spectrum of the vocal audio to be converted, and a preset loss function.

9. The method according to claim 8, characterized in that, The Diffusion-based generative model also includes: a discriminator; The method further includes: The Mel spectrum of the synthesized singing audio is input into the discriminator to obtain the predicted discrimination result output by the discriminator; The adversarial loss is calculated based on the actual identification result of the Mel spectrum of the synthesized singing audio and the predicted identification result. The Diffusion-based generative model is trained based on the reconstruction loss and backpropagation algorithm, including: The Diffusion-based generative model is trained based on the reconstruction loss, the adversarial loss, and the backpropagation algorithm.

10. A method for generating audio based on a singing voice conversion system, characterized in that, The singing conversion system includes at least a text encoder, a timbre encoder, a timbre-aware attention mechanism module, and a decoder; the method includes: Obtain multiple reference audios for the second target timbre, wherein the multiple reference audios for the second target timbre include at least multiple audios from different singers in different vocal registers; Multiple reference audios of the second target timbre are input to the timbre encoder to perform timbre encoding on the multiple reference audios of the second target timbre, thereby obtaining the timbre encoding vector of the multiple reference audios of the second target timbre output by the timbre encoder; The PPG phoneme posterior probability and the pitch fundamental frequency of the singing voice to be converted are input into the text encoder to obtain the prior distribution parameters of the singing voice content to be converted output by the text encoder. Based on the prior distribution parameters, samples are taken from the posterior probability of the PPG phonemes of the vocal to be converted and the fundamental frequency of the pitch of the vocal to be converted to obtain the text sample value vector of the vocal content to be converted. The text sampling value vector of the singing content to be converted and the timbre encoding vector of multiple reference audios of the second target timbre are input into the timbre perception attention mechanism module to determine a new timbre encoding vector based on the timbre encoding vector of multiple reference audios of the second target timbre and the text sampling value vector of the singing content to be converted. The text sample value vector of the singing content to be converted and the new timbre encoding vector are used as inputs to the decoder to obtain the synthesized singing audio.

11. The method according to claim 10, characterized in that, If the vocal conversion system includes the VITS vocal conversion system based on variational autoencoder, then the VITS vocal conversion system based on variational autoencoder also includes an inverse FLOW stream module. The method further includes: The text sample value vector of the singing content to be converted is input to the inverse FLOW stream module to map the simple distribution of the text sample value vector of the singing content to be converted to the complex distribution of audio, so as to obtain the mapped text sample value vector of the singing content to be converted. The step of using the new timbre encoding vector as input to the decoder to obtain the synthesized vocal audio includes: The text sample value vector of the mapped singing content to be converted and the new timbre encoding vector are concatenated or added to obtain the fused singing vector; The fused vocal vector is input into the decoder to obtain the synthesized vocal audio.

12. The method according to claim 10, characterized in that, If the singing conversion system includes the Fastspeeh fast speech synthesis model or the Diffusion diffusion generation model; The step of using the text sample value vector of the singing content to be converted and the new timbre encoding vector as input to the decoder to obtain the synthesized singing audio includes: The text sample value vector of the singing content to be converted and the new timbre encoding vector are concatenated or added to obtain the fused singing vector; The fused vocal vector is input into the decoder to obtain the synthesized vocal audio.

13. The method according to claim 11, characterized in that, If the singing conversion system includes the VITS singing conversion system based on variational autoencoder, then the text encoder includes a first multi-head attention mechanism module, and the decoder includes an upsampling module. If the singing conversion system includes the Fastspeeh fast speech synthesis model, then the text encoder includes a second multi-head attention mechanism module, and the decoder includes a third multi-head attention mechanism module; If the singing conversion system includes a Diffusion-based generation model, then the text encoder includes a fourth multi-head attention mechanism module, and the decoder includes a U-net module or a DiT module.

14. A computer device comprising a processor, characterized in that, When the processor executes a computer program stored in the memory, it is used to implement the training method of the singing voice conversion system as described in any one of claims 1 to 9, or the method of generating audio based on the improved singing voice conversion system as described in any one of claims 10 to 13.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it is used to implement the training method of the singing voice conversion system as described in any one of claims 1 to 9, or the method of generating audio based on the singing voice conversion system as described in any one of claims 10 to 13.

16. A computer program product having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it is used to implement the training method of the singing voice conversion system as described in any one of claims 1 to 9, or the method of generating audio based on the singing voice conversion system as described in any one of claims 10 to 13.

Images