Singing conversion method and device, electronic equipment, program product and storage medium

Abstract

The invention provides a singing sound conversion method and device, electronic equipment, a program product and a storage medium, and relates to the field of singing sound conversion, and the method comprises the steps: obtaining a basic sound audio and singing style information; encoding the basic stem sound audio into a basic stem sound vector, and encoding the singing style information into a singing style vector; using a trained diffusion model to carry out noise reduction processing on random noise according to the basic dry sound vector and the singing style vector to obtain a converted dry sound vector; decoding the converted dry sound vector to obtain a converted dry sound audio; on the basis of the noise reduction principle of the diffusion model, singing style conversion can be performed on the basic dry sound according to the singing style information, so that the flexibility of singing conversion can be improved.

Description

Technical Field

[0001] The present invention relates to the field of singing voice conversion, and in particular to a singing voice conversion method, device, electronic equipment, program product and storage medium. Background Art

[0002] Singing voice conversion refers to converting the input singing dry voice to obtain a converted dry voice with a specified timbre style while keeping the singing content unchanged. In the related art, singing voice conversion technology remains at the singing voice timbre conversion stage, that is, it can only convert the timbre of the input singing dry voice, but cannot convert the singing style of the singing dry voice. Summary of the invention

[0003] The purpose of the present invention is to provide a singing voice conversion method, device, electronic device, program product and storage medium, which can convert the basic dry voice into a singing style according to the singing style information based on the noise reduction principle of the diffusion model, thereby improving the flexibility of the singing voice conversion.

[0004] To solve the above technical problems, the present invention provides a singing voice conversion method, comprising:

[0005] Get basic dry voice audio and singing style information;

[0006] Encoding the basic dry voice audio into a basic dry voice vector, and encoding the singing style information into a singing style vector;

[0007] Using the trained diffusion model to perform noise reduction processing on random noise according to the basic dry voice vector and the singing style vector, to obtain a converted dry voice vector;

[0008] The converted dry sound vector is decoded to obtain the converted dry sound audio.

[0009] Optionally, encoding the basic dry sound audio into a basic dry sound vector includes:

[0010] Performing latent space encoding on the basic dry sound audio using a variational autoencoder to obtain the basic dry sound vector;

[0011] The step of using the trained diffusion model to perform noise reduction processing on random noise according to the basic dry sound vector and the singing style vector to obtain a converted dry sound vector includes:

[0012] Using the trained diffusion model to perform denoising on the random noise in a latent space according to the basic dry sound vector and the singing style vector, to obtain the converted dry sound vector;

[0013] The decoding of the converted dry sound vector to obtain the converted dry sound audio includes:

[0014] The converted dry sound vector is latently decoded by using a variational autoencoder to obtain the converted dry sound audio.

[0015] Optionally, the singing style information is singing style reference audio and / or singing style description text.

[0016] Optionally, encoding the singing style information into a singing style vector comprises:

[0017] The singing style information is encoded into the singing style vector using a multimodal pre-training model; wherein the multimodal pre-training model is used to encode the singing style reference audio and the singing style description text into the same feature space.

[0018] Optionally, the training process of the diffusion model includes:

[0019] Obtaining an initial diffusion model and multiple training dry sound audios; wherein different training dry sound audios have different timbres and styles;

[0020] Encoding the training dry sound audio into training basic dry sound vectors, training timbre vectors and training style vectors respectively, and performing multiple rounds of training processing on the initial diffusion model using the training basic dry sound vectors, the training timbre vectors and the training style vectors to obtain a basic diffusion model;

[0021] The trained diffusion model is obtained based on the basic diffusion model.

[0022] Optionally, the using the training basic dry sound vector, the training timbre vector and the training style vector to perform multiple rounds of training processing on the initial diffusion model to obtain a basic diffusion model includes:

[0023] When performing each round of training processing, different random noise signals are added to the training basic dry sound vector according to different noise step numbers to obtain multiple training noise feature maps; wherein the noise step number is used to control the noise intensity of the random noise;

[0024] Using the initial diffusion model to perform noise prediction on each of the training noise feature graphs according to the training timbre vector and the training style vector, to obtain a predicted noise signal corresponding to each of the noise feature graphs;

[0025] Determining a loss value between a random noise signal added to the training noise feature map and a predicted noise signal corresponding to the training noise feature map;

[0026] If the loss value is greater than a preset threshold, the model parameters of the initial diffusion model are updated according to the loss value, and the next round of training processing is entered;

[0027] If the loss value is not greater than a preset threshold, it is determined that the initial diffusion model has completed training, and the basic diffusion model is obtained.

[0028] Optionally, obtaining the trained diffusion model based on the basic diffusion model includes:

[0029] Obtain multiple copies of user dry audio;

[0030] The user dry voice audio is respectively encoded into a fine-tuned basic dry voice vector, a fine-tuned timbre vector and a fine-tuned style vector, and the basic diffusion model is trained multiple rounds using the fine-tuned basic dry voice vector, the fine-tuned timbre vector and the fine-tuned style vector to obtain the trained diffusion model.

[0031] Optionally, the performing multiple rounds of training processing on the basic diffusion model using the fine-tuned basic dry sound vector, the fine-tuned timbre vector, and the fine-tuned style vector to obtain the trained diffusion model includes:

[0032] The basic diffusion model is trained for multiple periods using the fine-tuned basic dry voice vector, the fine-tuned timbre vector and the fine-tuned style vector to obtain a trained diffusion model; wherein the basic diffusion model is trained using the fine-tuned basic dry voice vector, the fine-tuned timbre vector and the fine-tuned style vector of each user's dry voice audio in each fine-tuning period.

[0033] Optionally, obtaining the trained diffusion model based on the basic diffusion model includes:

[0034] using the basic diffusion model as the trained diffusion model;

[0035] The step of using the trained diffusion model to perform noise reduction processing on random noise according to the basic dry sound vector and the singing style vector to obtain a converted dry sound vector includes:

[0036] Acquire user timbre audio, and encode the user timbre audio into a user timbre vector;

[0037] The trained diffusion model is used to perform noise reduction processing on the random noise according to the basic dry sound vector, the user timbre vector, and the singing style vector to obtain the converted dry sound vector.

[0038] The present invention also provides a singing voice conversion device, comprising:

[0039] An acquisition module, used to acquire basic dry voice audio and singing style information;

[0040] An encoding module, used for encoding the basic dry voice audio into a basic dry voice vector, and encoding the singing style information into a singing style vector;

[0041] A conversion module, used for performing noise reduction processing on random noise according to the basic dry sound vector and the singing style vector by using a trained diffusion model to obtain a converted dry sound vector;

[0042] A decoding module is used to decode the converted dry sound vector to obtain the converted dry sound audio.

[0043] The present invention also provides an electronic device, comprising:

[0044] Memory for storing computer programs;

[0045] A processor is used to implement the singing voice conversion method as described above when executing the computer program.

[0046] The present invention also provides a computer program product, including a computer program or instructions, which implement the singing voice conversion method as described above when executed by a processor.

[0047] The present invention also provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are loaded and executed by a processor, the singing voice conversion method as described above is implemented.

[0048] The present application provides a singing voice conversion method, including: obtaining basic dry voice audio and singing style information; encoding the basic dry voice audio into a basic dry voice vector, and encoding the singing style information into a singing style vector; using a trained diffusion model to perform denoising on random noise according to the basic dry voice vector and the singing style vector to obtain a converted dry voice vector; decoding the converted dry voice vector to obtain a converted dry voice audio.

[0049] It can be seen that the present application can realize the singing style conversion of the singing dry voice based on the noise reduction capability of the trained diffusion model. Specifically, the present application can obtain basic dry voice audio and singing style information, the basic dry voice audio is used to provide basic singing content, and the singing style information is used to provide singing style. Subsequently, the present application can encode the basic dry voice audio into a basic dry voice vector, and encode the singing style information into a singing style vector, so as to convert the above audio and singing style information into a vector form that can be processed by the diffusion model. Furthermore, the present application can use the trained diffusion model to perform noise reduction processing on the random noise according to the basic dry voice vector and the singing style vector, and obtain the conversion dry voice vector, that is, based on the noise reduction principle of the diffusion model, the basic dry voice can be converted into a singing style according to the singing style. Finally, the present application can convert the dry voice vector for decoding, and obtain the conversion dry voice audio in which the singing content corresponds to the basic dry voice audio and the style corresponds to the singing style information, so that the singing dry voice can be flexibly converted into a singing style. The present application also provides a singing voice conversion device, an electronic device, a computer program product, and a computer-readable storage medium, which have the above-mentioned beneficial effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0051] Figure 1 A structural schematic diagram of a singing voice conversion system provided in an embodiment of the present application;

[0052] Figure 2 A flow chart of a singing voice conversion method provided in an embodiment of the present application;

[0053] Figure 3 A schematic diagram of a singing voice conversion process provided by an embodiment of the present application;

[0054] Figure 4 A schematic diagram of a model training process provided in an embodiment of the present application;

[0055] Figure 5 A schematic diagram of a noise adding process provided in an embodiment of the present application;

[0056] Figure 6 A schematic diagram of noise prediction provided in an embodiment of the present application;

[0057] Figure 7 A schematic diagram of loss value calculation provided in an embodiment of the present application;

[0058] Figure 8 A schematic diagram of another model training process provided in an embodiment of the present application;

[0059] Fig. 9 A schematic diagram of another singing voice conversion process provided by an embodiment of the present application;

[0060] Fig.10 A structural block diagram of a singing voice conversion device provided in an embodiment of the present application;

[0061] Fig.11 The present invention provides a structural block diagram of an electronic device. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0063] Singing voice conversion refers to converting the input singing dry voice while keeping the singing content unchanged, and obtaining a converted dry voice with a specified timbre style. Among them, the singing dry voice refers to the pure singing voice without accompaniment. In the related art, the singing voice conversion technology remains at the singing voice timbre conversion stage, that is, it can only convert the timbre of the input singing dry voice, but cannot convert the singing style of the singing dry voice. Among them, the timbre is the voice characteristics of the singer; the singing style refers to the style attributes such as pronunciation and singing skills. For example, for the singing dry voice of singer A, the related technology can only convert it into the timbre of singer B, but cannot convert it into the singing style of singer C. This is not conducive to the application of the singing voice conversion function.

[0064] In view of this, in order to address the technical problem that singing voice conversion can only perform timbre conversion but cannot perform singing style conversion, the present application can provide a singing voice conversion method, which can perform singing style conversion on the singing dry voice based on the noise reduction principle of the diffusion model according to the singing style information, thereby improving the flexibility of singing voice conversion.

[0065] First, the system structure applicable to this embodiment is introduced. Figure 1 , Figure 1This is a structural diagram of a singing voice conversion system provided in an embodiment of the present application. The system can be composed of multiple user terminal devices and servers, and data is transmitted between the two through a network. The user terminal device can send user data to the server, such as sending the basic dry voice specified by the user, the user's own dry voice, style information, etc.; in addition, a user interface can be set in the user terminal device to receive input from the user and display information to the user. The server can convert the singing voice according to the user data to obtain a converted dry voice, and send the converted dry voice to the user terminal device. That is, the server is the executor of the method provided by the present invention. The above-mentioned user terminal device can be a mobile device (such as a mobile phone, a tablet computer), a personal computer, etc. The above-mentioned server can be a server cluster, that is, multiple servers can interact with each user terminal device at the same time.

[0066] Furthermore, the diffusion model will be introduced below. The core operation of the diffusion model is noise reduction processing, which can gradually reduce the noise of an object consisting only of pure noise according to the given input information, and obtain a generated object that matches the input information. The modality of the object can be an image, a sound signal, etc. For example, in an image generation task, the diffusion model can gradually reduce the noise of a pure noise image according to the image generation requirements input by the user, and obtain a generated image that matches the image generation requirements. Since this noise reduction processing of the diffusion model can flexibly reconstruct the generated object specified by the user, it plays an important role in the singing style conversion of the singing dry voice in this application.

[0067] Of course, in addition to converting the singing style of the dry singing voice, the present application can also convert the timbre and singing style of the dry singing voice according to the given timbre information (such as user timbre) and singing style information based on the noise reduction principle of the diffusion model, and can further use the independently input timbre information and singing style information to adjust the timbre and style of the dry singing voice, thereby further improving the flexibility of the singing voice conversion.

[0068] It should be pointed out that the diffusion model can perform the above-mentioned noise reduction processing in different feature spaces, and different feature space selections will bring different performances. For example, for image generation tasks, the above-mentioned feature space can be a pixel feature space, that is, the diffusion model needs to reconstruct each pixel in the image. For the singing voice conversion task in this application, the above-mentioned feature space can be an audio feature space, that is, the diffusion model needs to reconstruct each frame of audio. However, the above two feature spaces consume too much computing resources. For higher-resolution images or audio with a longer duration, the diffusion model needs to spend more time and perform more noise reduction processing to obtain a better reconstruction effect. Therefore, the diffusion model can also select latent space as the feature space and perform noise reduction processing in the latent space. Among them, latent space is a representation of compressed data. The process of encoding the original modal data (such as image data, audio data) into a latent space vector is actually a compression and dimensionality reduction process of the original modal data. Therefore, the diffusion model performs noise reduction processing based on the latent space vector, which can significantly improve the processing efficiency.

[0069] The basic structure of the diffusion model includes: an input encoder, a generator, and an output decoder. The input encoder is used to encode the input information into a mathematical representation that can be processed by a computer; the generator is used to reduce the noise of the pure noise object into a mathematical representation of the generated object that matches the input information according to the mathematical representation; the output decoder is used to decode the mathematical representation of the generated object into a generated object. The generator is the core part of the diffusion model, and its basic framework can be composed of a UNet model and a sampler. Among them, the UNet model is a semantic segmentation model; the sampler can be DDPM (Denoising Diffusion Probabilistic Models, denoising diffusion probability model), DDIM (Denoising Diffusion Implicit Models, denoising diffusion implicit model), PLMS (Pseudo Linear Multi-Step Sampler, pseudo linear multi-step method), etc. Of course, for a diffusion model based on latent space for denoising, its sampler may have a more complex structure. The embodiment of the present application does not limit the specific structure of the diffusion model, and reference may be made to the relevant technologies of UNetModel, Stable Diffusion, and DiT (Diffusion Transformer). Based on the above system structure introduction and diffusion model introduction, the singing voice conversion method provided by the embodiment of the present application is described in detail below. Figure 2 , Figure 2 A flowchart of a singing voice conversion method provided in an embodiment of the present application, the method may include:

[0070] S101. Obtain basic dry voice audio and singing style information.

[0071] In this embodiment, the basic dry voice audio is the dry voice audio to be converted, which is used to provide basic singing content; the singing style information is used to provide the singing style, and the information can be either a singing style reference audio or a singing style description text, wherein the singing style reference audio is the dry voice audio providing the singing style, and the singing style description text describes the required singing style in text form. It should be pointed out that the basic dry voice audio needs to maintain a high dry voice quality, such as being in tune, clear in pronunciation, and free of noise, so the basic dry voice audio can generally be extracted from high-quality recorded music, such as extracting dry voice audio from existing music, or obtaining high-quality cover dry voice audio as the basic dry voice.

[0072] In addition, the diffusion model provided in this embodiment can also be used to convert the timbre of the basic dry voice audio, so in this step, additional timbre information can be obtained. This timbre information can be either live recorded or pre-recorded timbre audio, or a timbre vector pre-constructed using pre-collected timbre audio. For example, in order to enhance the fun of the singing voice conversion function and enhance the user's desire to record singing, the above-mentioned timbre information can be user timbre audio, which is a dry voice audio that provides user timbre information, and can be obtained by recording the user's own singing; the above-mentioned timbre information can also be a user timbre vector constructed using pre-recorded user timbre audio. It is worth pointing out that the user timbre audio only needs to provide the user's timbre, so there is no need to require a high dry voice quality. The user can record the user timbre audio in any language, and does not require the use of singing skills, even if it is out of tune, as long as the pronunciation is clear and there is no obvious noise. In other words, this embodiment does not have high requirements for the quality of the user timbre audio, which can lower the user's usage threshold.

[0073] S102: Encode the basic dry voice audio into a basic dry voice vector, and encode the singing style information into a singing style vector.

[0074] In this step, in order to facilitate the processing of the diffusion model, the basic dry voice audio and the singing style information need to be encoded into a representation vector form that can be processed by the diffusion model. The encoding methods of the basic dry voice audio and the singing style information are introduced below.

[0075] In this embodiment, the basic dry sound vector corresponding to the basic dry sound audio will be input into the diffusion model as the basic data to guide the model to perform multiple rounds of noise reduction. That is, logically, the basic dry sound audio is subjected to multiple rounds of conversion processing to obtain the converted dry sound audio whose timbre and style have been converted. As mentioned above, the diffusion model can perform noise reduction processing based on the audio feature space, or based on the latent space, and the model can achieve higher processing efficiency in the latter feature space. Therefore, in this embodiment, the basic dry sound audio can be encoded in the latent space to obtain the basic dry sound vector. In order to realize latent space encoding, this embodiment can introduce variational auto-encoders (VAE) for encoding. Variational auto-encoders will be briefly introduced here.

[0076] A variational autoencoder usually includes an encoder and a decoder. The encoder is used to extract discrete features of the input data and encode it into a continuous encoding vector based on the discrete features, while the decoder is used to decode the encoding vector and reconstruct the output data of the original mode. For example, a segment of audio is input into the above encoder to obtain a representation vector, and then the representation vector is input into the above decoder to obtain the restored audio. The clever feature of the variational autoencoder is that it uses some clever methods to constrain the encoding vector so that the encoding vector satisfies the standard normal distribution. In this way, the decoder not only recognizes the vectors encoded by the encoder, but also recognizes other vectors from the standard normal distribution. The training goal of the variational autoencoder is to make the reconstructed output data as similar as possible to the original input data. After the training is completed, the encoder and decoder in the variational autoencoder can work independently. In this embodiment, the encoder of the variational autoencoder is used to encode the basic dry sound audio into a basic dry sound vector.

[0077] Based on this, encoding the basic dry sound audio into a basic dry sound vector may include:

[0078] Step 11: Use a variational autoencoder to perform latent space encoding on the basic dry sound audio to obtain the basic dry sound vector.

[0079] Further, the singing style information in the present embodiment can be a singing style reference audio and / or a singing style description text. In this way, the user can input audio to indicate the singing style he needs, or use text to describe the singing style he needs. It is worth noting that although the singing style reference audio and the singing style description text are data of different modes, they can correspond to the same singing style features, that is, the singing style reference audio and the singing style description text can correspond to the same singing style features. Therefore, the present embodiment can introduce a multimodal pre-training model to encode the singing style reference audio and the singing style description text to the same feature space. The multimodal pre-training model is a model that can perform cross-modal feature mapping, for example, it can map audio and text to the same feature space, thereby connecting the two modes of audio and text. The model can include an audio encoder and a text encoder, the audio encoder is used to encode the singing style reference audio, and the text encoder is used to encode the singing style description text. The two encoders can be controlled based on a contrastive learning method to learn the corresponding relationship between the singing style reference audio and the singing style description text, thereby achieving the effect of mapping audio and text to the same feature space. It should be noted that this embodiment does not limit the structure of the multimodal pre-training model and the specific steps of the comparative learning method, and reference may be made to the relevant technologies of the multimodal pre-training model.

[0080] Based on this, encoding the singing style information into a singing style vector may include:

[0081] Step 21: Encode the singing style information into the singing style vector using a multimodal pre-training model, wherein the multimodal pre-training model is used to encode the singing style reference audio and the singing style description text into the same feature space.

[0082] Furthermore, if the user timbre audio is additionally obtained in the previous step, the user timbre audio may be encoded into a user timbre vector in this step to convert the user timbre audio into an encoding mode that can be processed by the model. In this embodiment, the timbre model (Timber Model) may be used to encode the user timbre audio into a user timbre vector. This embodiment does not limit the structure, encoding method, and training method of the timbre model, and reference may be made to the relevant technology of the model.

[0083] S103, using the trained diffusion model to perform noise reduction processing on random noise according to the basic dry voice vector and the singing style vector to obtain a converted dry voice vector.

[0084] After obtaining the basic dry voice vector and the singing style vector, the present embodiment can input them into the trained diffusion model to use the trained diffusion model to perform denoising on the random noise according to the basic dry voice vector and the singing style vector to obtain the converted dry voice vector. This denoising process can be performed multiple times to obtain the converted dry voice vector with high-quality conversion of the singing style. Furthermore, the denoising process can be performed in the latent space.

[0085] Based on this, the method of using the trained diffusion model to perform noise reduction processing on random noise according to the basic dry sound vector and the singing style vector to obtain a converted dry sound vector may include:

[0086] Step 31: Utilize the trained diffusion model to perform denoising on the random noise in latent space according to the basic dry sound vector and the singing style vector to obtain the converted dry sound vector.

[0087] It can be seen that, since this embodiment can reduce the dimension of the basic dry sound vector in the form of latent space coding, the processing efficiency of the diffusion model can be improved.

[0088] In addition, the diffusion model provided in this embodiment can also be used to convert the timbre of basic dry voice audio. Therefore, in this step, the basic dry voice vector, user timbre vector, and singing style vector can also be input into the diffusion model, and the trained diffusion model can be used to perform noise reduction processing on the random noise according to the basic dry voice vector, user timbre vector and singing style vector to obtain the converted dry voice vector, thereby obtaining the converted dry voice vector whose timbre and singing style have been converted.

[0089] Furthermore, the above-mentioned trained diffusion model can be either a basic diffusion model or a customized diffusion model, and can be set according to the actual application scenario. Among them, the basic diffusion model is trained using a number of training dry voice audios; the customized diffusion model is obtained by fine-tuning the training using the user's user dry voice audio on the basis of the basic diffusion model. For example, when the user does not need a customized diffusion model and needs to use the singing voice conversion function as soon as possible, the present embodiment can use the above-mentioned basic diffusion model to provide services to the user; when the user needs to use a higher quality singing voice conversion effect, the present embodiment can also obtain multiple segments of user dry voice audio from the user, such as requesting the user to upload or record 2-10 singing tracks, and use the user's dry voice audio to fine-tune the basic diffusion model, so as to use the customized diffusion model corresponding to the user to provide services to the user.

[0090] Furthermore, when using the basic diffusion model to provide services to users, if the user needs to convert the timbre of the basic dry audio to the user's timbre, this embodiment needs to obtain additional user timbre audio from the user, such as requesting the user to record a song. When using the customized diffusion model to provide services to users, if the user needs to convert the timbre of the basic dry audio to the user's timbre, this embodiment can use the pre-collected user dry audio as the user timbre audio.

[0091] S104: Decode the converted dry sound vector to obtain converted dry sound audio.

[0092] As described above, the diffusion model can be used to perform noise reduction based on the latent space, and the basic dry sound audio can be encoded in the latent space based on the variational autoencoder. Therefore, the transformed dry sound vector can also be decoded in the latent space using the variational autoencoder to obtain the transformed dry sound audio.

[0093] Based on this, decoding the converted dry sound vector to obtain the converted dry sound audio may include:

[0094] Step 41: Use a variational autoencoder to perform latent space decoding on the converted dry sound vector to obtain the converted dry sound audio.

[0095] Based on the above embodiments, the present application can realize the singing style conversion of the singing dry voice based on the noise reduction capability of the trained diffusion model. Specifically, the present application can obtain basic dry voice audio and singing style information, the basic dry voice audio is used to provide basic singing content, and the singing style information is used to provide singing style. Subsequently, the present application can encode the basic dry voice audio into a basic dry voice vector, and encode the singing style information into a singing style vector, so as to convert the above audio and singing style information into a vector form that can be processed by the diffusion model. Furthermore, the present application can use the trained diffusion model to perform noise reduction processing on random noise according to the basic dry voice vector and the singing style vector to obtain a converted dry voice vector, that is, based on the noise reduction principle of the diffusion model, the basic dry voice can be converted into a singing style according to the singing style. Finally, the present application can convert the dry voice vector for decoding, and obtain the converted dry voice audio in which the singing content corresponds to the basic dry voice audio and the style corresponds to the singing style information, so that the singing dry voice can be flexibly converted into a singing style.

[0096] Based on the above embodiment, for a comprehensive understanding of the above singing conversion process, please refer to Figure 3 , Figure 3A schematic diagram of a singing voice conversion process provided in an embodiment of the present application. In this schematic diagram, the basic dry voice audio can be input into a latent space encoder (i.e., a variational autoencoder) for encoding to obtain a basic dry voice vector, the user timbre audio can be input into a timbre model to obtain a user timbre vector, and the singing style reference audio can be input into an audio encoder (i.e., the audio branch of a multimodal pre-training model) to obtain a singing style vector. Optionally, the singing style description text can also be input into a text encoder (i.e., the text branch of a multimodal pre-training model) to obtain a singing style vector. Subsequently, the basic dry voice vector, the user timbre vector, and the singing style vector can all be input into a diffusion model for noise reduction processing to obtain a converted dry voice vector, and the converted dry voice vector can be input into a latent space decoder (i.e., a variational autoencoder) for decoding to obtain a converted dry voice audio.

[0097] Based on the above embodiment, the training process of the basic diffusion model is introduced below. The training process of the diffusion model may include:

[0098] S201, obtaining an initial diffusion model and a plurality of training dry sound audios; wherein different training dry sound audios have different timbres and styles.

[0099] In this step, the initial diffusion model and multiple training dry audios are first obtained. Among them, the initial diffusion model refers to a diffusion model that has not undergone any training. The training dry audio is high-quality dry audio, which is used to train the initial diffusion model to obtain a basic diffusion model. Since different training dry audios carry different timbre and style information, the initial diffusion model can learn the relevant knowledge of timbre and style conversion after being exposed to a large amount of training dry audio. It is worth noting that the above training dry audio does not need to be annotated with music scores, phoneme boundaries, timbre, style and other annotation information, which can effectively lower the training threshold.

[0100] It should be noted that the present embodiment does not limit the number of training dry audio, which can be set according to actual application requirements. It is understandable that the more the number of training dry audio, the more timbre and style types the initial diffusion model can be exposed to during the training process, and the stronger the ability to convert timbre and style. It should also be pointed out that in order to avoid a single training dry audio being too long, it can be segmented according to a preset duration to obtain multiple segments for subsequent training based on the segments. The present embodiment does not limit the specific value of the preset duration, for example, it can be 5 seconds.

[0101] S202. Encode the training dry sound audio into training basic dry sound vectors, training timbre vectors and training style vectors respectively, and use the training basic dry sound vectors, the training timbre vectors and the training style vectors to perform multiple rounds of training processing on the initial diffusion model to obtain a basic diffusion model.

[0102] This step will use the training dry voice audio to train the initial diffusion model. First, this step can encode the training dry voice audio into training basic dry voice vectors, training timbre vectors and training style vectors respectively. In order to facilitate the use of training data, the three vectors generated by the same training dry voice audio can be formed into a triplet (basic dry voice vector, training timbre vector, training style vector), and multiple triplets corresponding to multiple training dry voice audios are obtained. These triplets can be used to train the basic diffusion model later. It should be noted that the role and encoding method of the training basic dry voice vector, training timbre vector and training style vector are the same as the above-mentioned basic dry voice vector, user timbre vector and singing style vector, which will not be repeated here. It should also be noted that the variational autoencoder, timbre model and multimodal pre-trained language model used for encoding here have been trained, and this embodiment only trains the diffusion model part.

[0103] This step can then use the above-mentioned training basic dry sound vector, training timbre vector and training style vector to train the initial diffusion model. It should be noted that the training basic dry sound vector, training timbre vector and training style vector used in the same round of training are generated by the same training dry sound audio. This is because the training dry sound audio will not be labeled in the present embodiment first; and, unlike the process of denoising random noise in the inference stage, the initial diffusion model learns the process of adding noise to the training basic dry sound vector, which can be briefly described as:

[0104] 1. Add random noise to the training basic dry sound vector to obtain the noise feature map;

[0105] 2. Input the noise feature map, the training timbre vector, and the training style vector into the initial diffusion model to control the initial diffusion model to perform noise prediction and obtain predicted noise;

[0106] 3. Determine the difference between the random noise and the predicted noise, determine the loss value, and use the loss value to update the initial diffusion model.

[0107] Under ideal conditions, the training basic dry voice vector can be obtained after removing the predicted noise signal from the noise feature map. Therefore, the above process is logically equivalent to controlling the model to restore the training basic dry voice vector from the training noise feature map according to the timbre information in the training timbre vector and the style information in the training style vector. In order to constrain the initial diffusion model to accurately predict the noise, so that the noise feature map minus the predicted noise can obtain a training basic dry voice vector with a specific timbre and a specific style, the timbre corresponding to the training timbre vector must be consistent with the timbre corresponding to the training basic dry voice vector, and the style corresponding to the training style vector must be consistent with the timbre corresponding to the training basic dry voice vector. If the training basic dry voice vector, training timbre vector and training style vector used in the same round of training are generated by different training dry voice audio, the timbre corresponding to the training timbre vector must be inconsistent with the timbre corresponding to the training basic dry voice vector, and the style corresponding to the training style vector must be inconsistent with the timbre corresponding to the training basic dry voice vector. In this case, the accuracy of the predicted noise cannot be determined, and the effectiveness of the model training cannot be judged. Therefore, the training basic dry sound vector, training timbre vector and training style vector used in the same round of training are generated by the same training dry sound audio. To facilitate training, the above triplets can be used one by one to train the basic diffusion model.

[0108] Furthermore, in the above process, the more accurate the predicted noise signal is, the stronger the model's ability to convert timbre and singing style is. Therefore, after obtaining the predicted noise signal, the present application can calculate the difference between the predicted noise signal and the added random noise signal, and use the difference as the loss value to update the model parameters of the initial diffusion model, so that the noise signal predicted by the model is more accurate. It should be noted that this embodiment does not limit how to calculate the above loss value, and reference may be made to the relevant technology of the neural network model.

[0109] Furthermore, since the denoising process of the diffusion model can be performed in multiple steps to gradually reduce the noise signal in the converted dry sound vector. Therefore, in order to ensure that the diffusion model can learn this gradual denoising process, the noise step number can also be introduced when constructing the training noise feature map. The noise step number can be used to control the noise intensity of the random noise, so as to add different random noise signals to the training basic dry sound vector according to different noise step numbers, and obtain multiple training noise feature maps. Furthermore, the present application can input the training noise feature map, noise step number, training timbre vector and training style vector into the initial diffusion model, so that the initial diffusion model predicts the noise signal in the training noise feature map according to the training timbre vector, training style vector and noise step number, and obtains predicted noise signals of different intensities.

[0110] Finally, after calculating the loss value between the predicted noise signal and the added random noise signal, the present embodiment can determine whether to perform the next round of training processing on the initial diffusion model based on the relationship between the loss value and the preset threshold. If the loss value is greater than the preset threshold, the model parameters of the initial diffusion model are updated according to the loss value, and the next round of training processing is entered; if the loss value is not greater than the preset threshold, it is determined that the initial diffusion model has completed training and a basic diffusion model is obtained.

[0111] Based on this, the initial diffusion model is trained multiple times using the training basic dry sound vector, the training timbre vector and the training style vector to obtain a basic diffusion model, including:

[0112] Step 51: When performing each round of training processing, adding a random noise signal to the training basic dry sound vector according to different noise step numbers to obtain multiple training noise feature maps; wherein the noise step number is used to control the noise intensity of the random noise;

[0113] Step 52: using the initial diffusion model to perform noise prediction on each of the training noise feature graphs according to the training timbre vector and the training style vector, to obtain a predicted noise signal corresponding to each of the noise feature graphs;

[0114] Step 53: determining a loss value between the random noise signal added to the training noise feature graph and the predicted noise signal corresponding to the training noise feature graph;

[0115] Step 54: if the loss value is greater than a preset threshold, the model parameters of the initial diffusion model are updated according to the loss value, and the next round of training processing is entered;

[0116] Step 55: If the loss value is not greater than a preset threshold, it is determined that the initial diffusion model has completed training, and the basic diffusion model is obtained.

[0117] It should be noted that this embodiment does not limit the specific value of the preset threshold, which can be set according to actual application requirements.

[0118] For a comprehensive understanding of the initial model training process, please refer to Figures 4 to 7 , Figure 4A schematic diagram of a model training process provided in an embodiment of the present application. In this schematic diagram, the training dry voice audio can be respectively input into a latent space encoder (i.e., a variational autoencoder), a timbre model, and an audio encoder (i.e., an audio branch of a multimodal pre-training model) for encoding to obtain a training basic dry voice vector, a training timbre vector, and a training style vector. Optionally, a singing style description text can also be input into a text encoder to obtain another training style vector. Subsequently, the training basic dry voice vector, the training timbre vector, and the training style vector can be input into a diffusion model for training. Figure 5 A schematic diagram of a noise adding process provided in an embodiment of the present application. The training basic dry sound vector, random noise, and noise step number can be input into the scheduler, so that the scheduler adds different random noise signals to the training basic dry sound vector according to different noise step numbers to obtain multiple training noise feature maps. Figure 6 A schematic diagram of noise prediction provided in an embodiment of the present application is provided. A noise feature map, a training timbre vector, a training style vector and a noise step number can be input into a diffusion model to obtain predicted noise. Figure 7 A schematic diagram of loss value calculation provided in an embodiment of the present application, wherein the predicted noise and the added random noise can be input into a preset evaluation index to calculate the loss value, so as to use the loss value to update the model parameters. It should be noted that this embodiment does not limit the specific form of the evaluation index, and it can be set according to actual application requirements.

[0119] S203: Obtain the trained diffusion model based on the basic diffusion model.

[0120] As described above, this embodiment can directly use the basic diffusion model as the trained diffusion model, and can also use the user's dry sound data to continue fine-tuning training on the basis of the basic diffusion model to obtain a customized diffusion model, and use the customized diffusion model as the trained diffusion model, which can be set according to actual application requirements.

[0121] Based on the above embodiment, the fine-tuning process of the basic diffusion model is introduced below. Based on this, the step of obtaining the trained diffusion model based on the basic diffusion model may include:

[0122] S301. Acquire multiple copies of user dry audio.

[0123] In this step, the user dry audio corresponds to a specific user and is used to fine-tune the basic diffusion model. It is worth noting that the above user dry audio also does not need to be annotated with music scores, phoneme boundaries, timbre, style and other annotation information, which can effectively reduce the training threshold.

[0124] It should be noted that the present embodiment does not limit the number of user dry voice audio, which can be set according to actual application requirements. It is also worth pointing out that since the basic diffusion model has been trained with a large amount of data, the demand for user dry voice audio is relatively small during the fine-tuning training process, that is, the present application only needs to use a small amount of user dry voice audio to fine-tune a diffusion model corresponding to a specific user. S302. Encode the user dry voice audio into a fine-tuned basic dry voice vector, a fine-tuned timbre vector, and a fine-tuned style vector, respectively, and use the fine-tuned basic dry voice vector, the fine-tuned timbre vector, and the fine-tuned style vector to perform multiple rounds of training processing on the basic diffusion model to obtain a trained diffusion model.

[0125] This step can encode the user's dry voice audio into a fine-tuned basic dry voice vector, a fine-tuned timbre vector and a fine-tuned style vector respectively. The functions and encoding methods of the fine-tuned basic dry voice vector, the fine-tuned timbre vector and the fine-tuned style vector are the same as those of the above-mentioned basic dry voice vector, the fine-tuned timbre vector and the singing style vector, and will not be repeated here. Similar to step S202, it is also necessary to input the fine-tuned basic dry voice vector, the fine-tuned timbre vector and the fine-tuned style vector into the basic diffusion model for training. It is worth pointing out that the trained diffusion model can be obtained by only using the fine-tuned basic dry voice vector, the fine-tuned timbre vector and the fine-tuned style vector to train the basic diffusion model for a preset number of epochs. Among them, the basic diffusion model uses the fine-tuned basic dry voice vector, the fine-tuned timbre vector and the fine-tuned style vector of each user's dry voice audio for training in each fine-tuning period.

[0126] Based on this, the basic diffusion model is trained multiple times using the fine-tuned basic dry sound vector, the fine-tuned timbre vector, and the fine-tuned style vector to obtain a trained diffusion model, including:

[0127] Step 61: Use the fine-tuned basic dry voice vector, the fine-tuned timbre vector and the fine-tuned style vector to train the basic diffusion model for multiple periods to obtain a trained diffusion model; wherein the basic diffusion model is trained using the fine-tuned basic dry voice vector, the fine-tuned timbre vector and the fine-tuned style vector of each user's dry voice audio in each fine-tuning period.

[0128] To fully understand the training process of the above custom diffusion model, please refer to Figure 8 , Figure 8A schematic diagram of another model training process provided in an embodiment of the present application. In this schematic diagram, the user's dry voice audio can be input into a latent space encoder (i.e., a variational autoencoder), a timbre model, and an audio encoder (i.e., an audio branch of a multimodal pre-training model) for encoding to obtain a fine-tuned basic dry voice vector, a fine-tuned timbre vector, and a fine-tuned style vector. Optionally, a singing style description text can also be input into a text encoder to obtain another fine-tuned style vector. Subsequently, the fine-tuned basic dry voice vector, the fine-tuned timbre vector, and the fine-tuned style vector can be input into a diffusion model for fine-tuning.

[0129] Finally, please refer to Fig. 9 , Fig. 9 This is a schematic diagram of another singing voice conversion process provided by an embodiment of the present application. In this embodiment, a dry voice material can be obtained first, and the dry voice material can be segmented to obtain multiple small segments; then, the small segments are encoded, feature extracted, and the model is fine-tuned and trained. Finally, a basic dry voice can be obtained, and the basic dry voice can be input into the trained model for processing to obtain a converted dry voice after timbre and style conversion.

[0130] Based on the above embodiment, the specific process of providing singing voice tone and style conversion to the user when the customized diffusion model is used as the trained diffusion model is introduced below. In a possible case, the singing voice conversion method provided in this embodiment may also include:

[0131] S401, obtaining basic dry voice audio and singing style information.

[0132] S402: Encode the basic dry voice audio into a basic dry voice vector, and encode the singing style information into a singing style vector.

[0133] S403, using the trained diffusion model to perform noise reduction processing on random noise according to the basic dry voice vector, the user timbre vector and the singing style vector to obtain a converted dry voice vector; the user timbre vector is obtained by encoding the user dry voice audio.

[0134] S404: Decode the converted dry sound vector to obtain converted dry sound audio.

[0135] In this embodiment, since the user dry voice audio is collected in advance to generate a customized diffusion model, when using the customized diffusion model to convert the basic dry voice audio into the user timbre, there is no need to obtain the user timbre audio from the user additionally. It is only necessary to use the pre-collected user dry voice audio to generate the user timbre vector, and input the user timbre vector together with the basic dry voice vector and the singing style vector into the diffusion model.

[0136] Based on the above embodiment, the specific process of providing the user with singing voice tone and style conversion when the basic diffusion model is used as the trained diffusion model is introduced below. In a possible case, the singing voice conversion method provided in this embodiment may also include:

[0137] S501, obtaining basic dry voice audio, user timbre audio and singing style information.

[0138] In this embodiment, since the basic diffusion model is used to provide services to the user, when the user needs to convert the basic dry sound audio into the user timbre, the user timbre audio also needs to be acquired.

[0139] S502. Encode the basic dry voice audio into a basic dry voice vector, encode the user timbre audio into a user timbre vector, and encode the singing style information into a singing style vector.

[0140] S503, using the trained diffusion model to perform noise reduction processing on random noise according to the basic dry sound vector, the user timbre vector and the singing style vector to obtain a converted dry sound vector.

[0141] S504: Decode the converted dry sound vector to obtain converted dry sound audio.

[0142] In this embodiment, the basic dry voice vector, user timbre vector and singing style vector can all be input into the diffusion model, so that the diffusion model can be used to perform noise reduction on the random noise according to the basic dry voice vector, user timbre vector and singing style vector, and the obtained converted dry voice vector can be decoded to obtain the converted dry voice audio in which the timbre and singing style have been converted.

[0143] The singing voice conversion device, electronic device, computer program product, and computer-readable storage medium provided in the embodiments of the present invention are introduced below. The singing voice conversion device, electronic device, computer program product, and computer-readable storage medium described below can be referenced to the singing voice conversion method described above.

[0144] Please refer to Fig.10 , Fig.10 A structural block diagram of a singing voice conversion device provided in an embodiment of the present application, the device may include:

[0145] An acquisition module 1001 is used to acquire basic dry voice audio and singing style information;

[0146] An encoding module 1002, configured to encode the basic dry voice audio into a basic dry voice vector, and encode the singing style information into a singing style vector;

[0147] A conversion module 1003 is used to perform noise reduction processing on random noise according to the basic dry sound vector and the singing style vector using a trained diffusion model to obtain a converted dry sound vector;

[0148] The decoding module 1004 is used to decode the converted dry sound vector to obtain the converted dry sound audio.

[0149] Optionally, the encoding module 1002 includes:

[0150] A first encoding submodule, configured to perform latent space encoding on the basic dry sound audio by using a variational autoencoder to obtain the basic dry sound vector;

[0151] The conversion module 1003 may be used to:

[0152] Using the trained diffusion model to perform denoising on the random noise in a latent space according to the basic dry sound vector and the singing style vector, to obtain the converted dry sound vector;

[0153] The decoding module 1004 may be used to:

[0154] The converted dry sound vector is latently decoded by using a variational autoencoder to obtain the converted dry sound audio.

[0155] Optionally, the encoding module 1002 may include:

[0156] The second encoding submodule is used to encode the singing style information into the singing style vector using a multimodal pre-training model, wherein the multimodal pre-training model is used to encode the singing style reference audio and the singing style description text into the same feature space.

[0157] Optionally, the device may further include:

[0158] A training data acquisition module, used to acquire an initial diffusion model and multiple training dry audios; wherein different training dry audios have different timbres and styles;

[0159] A training module, used for encoding the training dry sound audio into training basic dry sound vectors, training timbre vectors and training style vectors respectively, and performing multiple rounds of training processing on the initial diffusion model using the training basic dry sound vectors, the training timbre vectors and the training style vectors to obtain a basic diffusion model;

[0160] A model setting module is used to obtain the trained diffusion model based on the basic diffusion model.

[0161] Optionally, the first training module includes:

[0162] A noise signal adding submodule, used to add a random noise signal to the training basic dry sound vector according to different noise step numbers during each round of training processing, so as to obtain multiple training noise feature maps; wherein the noise step number is used to control the noise intensity of the random noise;

[0163] A noise prediction submodule, configured to use the initial diffusion model to perform noise prediction on each of the training noise feature graphs according to the training timbre vector and the training style vector, so as to obtain a predicted noise signal corresponding to each of the noise feature graphs;

[0164] A loss value calculation submodule, used to determine a loss value between a random noise signal added to the training noise feature map and a predicted noise signal corresponding to the training noise feature map;

[0165] A loop control submodule, configured to update the model parameters of the initial diffusion model according to the loss value and enter the next round of training processing if the loss value is greater than a preset threshold;

[0166] The training completion determination submodule is used to determine that the initial diffusion model has completed training if the loss value is not greater than a preset threshold, so as to obtain the basic diffusion model.

[0167] Optionally, the model setting module may include:

[0168] A user dry voice acquisition submodule is used to acquire multiple copies of user dry voice audio;

[0169] The model fine-tuning submodule is used to encode the user dry voice audio into a fine-tuned basic dry voice vector, a fine-tuned timbre vector and a fine-tuned style vector, respectively, and use the fine-tuned basic dry voice vector, the fine-tuned timbre vector and the fine-tuned style vector to perform multiple rounds of training processing on the basic diffusion model to obtain the trained diffusion model.

[0170] Optionally, the model fine-tuning submodule is specifically used to:

[0171] The basic diffusion model is trained for multiple periods using the fine-tuned basic dry voice vector, the fine-tuned timbre vector and the fine-tuned style vector to obtain a trained diffusion model; wherein the basic diffusion model is trained using the fine-tuned basic dry voice vector, the fine-tuned timbre vector and the fine-tuned style vector of each user's dry voice audio in each fine-tuning period.

[0172] Optionally, the model setting module may include:

[0173] A model setting submodule, used for using the basic diffusion model as the trained diffusion model;

[0174] The acquisition module 1001 may also be used for:

[0175] Get the user's audio.

[0176] The encoding module 1002 may also be used for:

[0177] Encoding the user timbre audio into a user timbre vector;

[0178] The conversion module 1003 may also be used for:

[0179] The trained diffusion model is used to perform noise reduction processing on the random noise according to the basic dry sound vector, the user timbre vector, and the singing style vector to obtain the converted dry sound vector.

[0180] Please refer to Fig.11 , Fig.11 This is a structural block diagram of an electronic device provided in an embodiment of the present invention. The embodiment of the present invention provides an electronic device 40, including a processor 41 and a memory 42; wherein the memory 42 is used to store a computer program; and the processor 41 is used to execute the singing voice conversion method provided in the aforementioned embodiment when executing the computer program.

[0181] For the specific process of the above-mentioned singing voice conversion method, please refer to the corresponding content provided in the aforementioned embodiment, which will not be repeated here.

[0182] Furthermore, the memory 42 as a carrier for storing resources may be a read-only memory, a random access memory, a magnetic disk or an optical disk, etc., and the storage method may be temporary storage or permanent storage.

[0183] In addition, the electronic device 40 also includes a power supply 43, a communication interface 44, an input / output interface 45 and a communication bus 46; wherein the power supply 43 is used to provide working voltage for each hardware device on the electronic device 40; the communication interface 44 can create a data transmission channel between the electronic device 40 and external devices, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present invention, and is not specifically limited here; the input / output interface 45 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0184] An embodiment of the present invention further provides a computer program product, including a computer program / instruction, which implements the singing voice conversion method described in the above embodiment when executed by a processor.

[0185] Since the embodiments of the computer program product part correspond to the embodiments of the singing voice conversion method part, please refer to the description of the embodiments of the singing voice conversion method part for the embodiments of the computer program product part, and will not be repeated here.

[0186] The embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the singing voice conversion method described in the above embodiment is implemented.

[0187] Since the embodiments of the computer-readable storage medium part correspond to the embodiments of the singing voice conversion method part, please refer to the description of the embodiments of the singing voice conversion method part for the embodiments of the storage medium part, which will not be repeated here.

[0188] The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0189] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0190] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0191] The singing voice conversion method, device, electronic device, program product and storage medium provided by the present invention are introduced in detail above. Specific examples are used herein to illustrate the principle and implementation mode of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. It should be pointed out that for ordinary technicians in this technical field, without departing from the principle of the present invention, the present invention can also be improved and modified, and these improvements and modifications also fall within the scope of protection of the present invention.

Claims

1. A singing voice conversion method, characterized in that: include: Get basic dry voice audio and singing style information; Encoding the basic dry voice audio into a basic dry voice vector, and encoding the singing style information into a singing style vector; Using the trained diffusion model to perform noise reduction processing on random noise according to the basic dry voice vector and the singing style vector, to obtain a converted dry voice vector; The converted dry sound vector is decoded to obtain the converted dry sound audio.

2. The singing voice conversion method according to claim 1, characterized in that: The step of encoding the basic dry sound audio into a basic dry sound vector comprises: Performing latent space encoding on the basic dry sound audio using a variational autoencoder to obtain the basic dry sound vector; The step of using the trained diffusion model to perform noise reduction processing on random noise according to the basic dry sound vector and the singing style vector to obtain a converted dry sound vector includes: Using the trained diffusion model to perform denoising on the random noise in a latent space according to the basic dry sound vector and the singing style vector, to obtain the converted dry sound vector; The decoding of the converted dry sound vector to obtain the converted dry sound audio includes: The converted dry sound vector is latently decoded by using a variational autoencoder to obtain the converted dry sound audio.

3. The singing voice conversion method according to claim 1, characterized in that: The singing style information is singing style reference audio and / or singing style description text.

4. The singing voice conversion method according to claim 3, characterized in that: The step of encoding the singing style information into a singing style vector comprises: The singing style information is encoded into the singing style vector using a multimodal pre-training model; wherein the multimodal pre-training model is used to encode the singing style reference audio and the singing style description text into the same feature space.

5. The singing voice conversion method according to any one of claims 1 to 4, characterized in that: The training process of the diffusion model includes: Obtaining an initial diffusion model and multiple training dry sound audios; wherein different training dry sound audios have different timbres and styles; Encoding the training dry sound audio into training basic dry sound vectors, training timbre vectors and training style vectors respectively, and performing multiple rounds of training processing on the initial diffusion model using the training basic dry sound vectors, the training timbre vectors and the training style vectors to obtain a basic diffusion model; The trained diffusion model is obtained based on the basic diffusion model.

6. The singing voice conversion method according to claim 5, characterized in that: The method of performing multiple rounds of training processing on the initial diffusion model using the training basic dry sound vector, the training timbre vector, and the training style vector to obtain a basic diffusion model includes: When performing each round of training processing, a random noise signal is added to the training basic dry sound vector according to different noise step numbers to obtain multiple training noise feature maps; wherein the noise step number is used to control the noise intensity of the random noise; Using the initial diffusion model to perform noise prediction on each of the training noise feature graphs according to the training timbre vector and the training style vector, to obtain a predicted noise signal corresponding to each of the noise feature graphs; Determining a loss value between a random noise signal added to the training noise feature map and a predicted noise signal corresponding to the training noise feature map; If the loss value is greater than a preset threshold, the model parameters of the initial diffusion model are updated according to the loss value, and the next round of training processing is entered; If the loss value is not greater than a preset threshold, it is determined that the initial diffusion model has completed training, and the basic diffusion model is obtained.

7. The singing voice conversion method according to claim 5, characterized in that: The step of obtaining the trained diffusion model based on the basic diffusion model includes: Obtain multiple copies of user dry audio; The user dry voice audio is respectively encoded into a fine-tuned basic dry voice vector, a fine-tuned timbre vector and a fine-tuned style vector, and the basic diffusion model is trained multiple rounds using the fine-tuned basic dry voice vector, the fine-tuned timbre vector and the fine-tuned style vector to obtain the trained diffusion model.

8. The singing voice conversion method according to claim 7, characterized in that: The step of performing multiple rounds of training processing on the basic diffusion model using the fine-tuned basic dry sound vector, the fine-tuned timbre vector, and the fine-tuned style vector to obtain the trained diffusion model includes: The basic diffusion model is trained for multiple periods using the fine-tuned basic dry voice vector, the fine-tuned timbre vector and the fine-tuned style vector to obtain a trained diffusion model; wherein the basic diffusion model is trained using the fine-tuned basic dry voice vector, the fine-tuned timbre vector and the fine-tuned style vector of each user's dry voice audio in each fine-tuning period.

9. The singing voice conversion method according to claim 5, characterized in that: The step of obtaining the trained diffusion model based on the basic diffusion model includes: using the basic diffusion model as the trained diffusion model; The step of using the trained diffusion model to perform noise reduction processing on random noise according to the basic dry sound vector and the singing style vector to obtain a converted dry sound vector includes: Acquire user timbre audio, and encode the user timbre audio into a user timbre vector; The trained diffusion model is used to perform noise reduction processing on the random noise according to the basic dry sound vector, the user timbre vector, and the singing style vector to obtain the converted dry sound vector.

10. A singing voice conversion device, characterized in that: include: An acquisition module, used to acquire basic dry voice audio and singing style information; An encoding module, used for encoding the basic dry voice audio into a basic dry voice vector, and encoding the singing style information into a singing style vector; A conversion module, used for performing noise reduction processing on random noise according to the basic dry sound vector and the singing style vector by using a trained diffusion model to obtain a converted dry sound vector; A decoding module is used to decode the converted dry sound vector to obtain the converted dry sound audio.

11. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, used to implement the singing voice conversion method as described in any one of claims 1 to 9 when executing the computer program.

12. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the singing voice conversion method described in any one of claims 1 to 9 is implemented.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are loaded and executed by the processor, the singing voice conversion method as described in any one of claims 1 to 9 is implemented.

Images