Song conversion model training method and device, equipment, medium and product

Abstract

This application provides a training method, apparatus, device, medium, and product for a singing voice conversion model. The method includes: acquiring initial speech data from a user; wherein the initial speech data represents the sound wave signal of a sentence of a preset length; obtaining a preset number of cloned speech data based on the user's initial speech data and a preset large language model; wherein the cloned speech data are speech data similar to the initial speech data; training the initial model based on the preset number of cloned speech data to obtain a trained singing voice conversion model; wherein the singing voice conversion model is used to convert the singing voice of preset content into the user's singing voice. The method of this application saves the cost of training the singing voice conversion model and improves the accuracy of the model's singing voice conversion.

Description

Technical Field

[0001] This application relates to the field of audio technology, and in particular to a training method, apparatus, device, medium and product for a singing voice conversion model. Background Technology

[0002] A voice conversion model can be used to convert one person's singing voice into another person's singing voice.

[0003] Training a singing voice conversion model typically requires acquiring a large amount of high-quality speech data to ensure good accuracy. However, acquiring large amounts of high-quality data is costly.

[0004] Therefore, training a singing voice conversion model faces the problem of high training costs or high data processing costs. Summary of the Invention

[0005] This application provides a training method, apparatus, device, medium, and product for a singing voice conversion model, to solve the technical problems of high training cost or high data processing cost when training a singing voice conversion model.

[0006] Firstly, this application provides a training method for a singing voice conversion model, comprising:

[0007] Acquire the user's initial voice data; wherein, the initial voice data represents the sound wave signal of a sentence of a preset length;

[0008] Based on the user's initial voice data and a preset large language model, a preset number of cloned voice data are obtained; among them, cloned voice data are voice data similar to the initial voice data.

[0009] The initial model is trained based on a preset number of cloned voice data to obtain a trained singing voice conversion model; the singing voice conversion model is used to convert the singing voice of the preset content into the user's singing voice.

[0010] Optionally, as described above, the initial model is trained using a preset number of cloned speech data to obtain a trained singing conversion model, including:

[0011] Feature extraction processing is performed on each cloned speech data in a preset number of cloned speech data to obtain feature information of each cloned speech data; wherein, the feature information includes at least one of music theory features, frequency domain features, volume features, text features, audio autoencoder features, and voiceprint features.

[0012] Based on the feature information of a preset number of cloned speech data and preset noise features, the initial model is trained to obtain a trained singing conversion model.

[0013] Optionally, as described above, feature extraction processing is performed on each cloned speech data in a preset number of cloned speech data to obtain feature information for each cloned speech data, including:

[0014] Each cloned speech data in a preset number of cloned speech data is preprocessed to obtain target speech data; wherein, the target speech data represents the preprocessed cloned speech data, and the preprocessing includes noise reduction processing and / or speech restoration processing;

[0015] Feature extraction is performed on the target speech data to obtain the feature information of the cloned speech data.

[0016] Optionally, as described above, the feature information includes music theory features; feature extraction processing is performed on the target speech data to obtain the feature information of the cloned speech data, including:

[0017] Based on a preset low-pass filter, the target speech data is filtered to obtain the music theory features of the cloned speech data.

[0018] Optionally, as described above, the feature information includes frequency domain features; feature extraction processing is performed on the target speech data to obtain the feature information of the cloned speech data, including:

[0019] Based on a preset Mel filter, the target speech data is processed by Mel spectrum filtering to obtain the frequency domain features of the cloned speech data.

[0020] Optionally, as described above, the feature information includes volume features; feature extraction processing is performed on the target speech data to obtain the feature information of the cloned speech data, including:

[0021] The target speech data is segmented to obtain multiple frame signals; each frame signal is a part of the target speech data and all frame signals have the same length.

[0022] The sum of squared amplitudes of each frame of signal is determined, and the sums of squared amplitudes of each frame of signal are added together to obtain the volume characteristics of the cloned speech data; where the sum of squared amplitudes represents the power spectral density.

[0023] Optionally, as described above, the initial model is trained based on the feature information of a preset number of cloned speech data and preset noise features to obtain a trained singing conversion model, including:

[0024] The feature information of each cloned speech data in a preset number of cloned speech data is fused with preset noise features to obtain target features; wherein, the target features represent the feature information of the cloned speech data and the preset noise features.

[0025] Acquire pre-collected training speech data, and train the initial model based on the training speech data, target features, and cloned speech data to obtain a trained singing conversion model; wherein, the training speech data is non-user speech data.

[0026] Optionally, as described above, the initial model is trained based on the speech data to be trained, the target features, and the cloned speech data to obtain a trained singing conversion model, including:

[0027] The speech data to be trained and the target features are input into the initial model, and the predicted singing information is output.

[0028] If the similarity between the predicted singing information and the cloned speech data meets a preset threshold, then the trained singing conversion model is obtained.

[0029] Optionally, as described above, based on the user's initial voice data and a preset large language model, a preset number of cloned voice data are obtained, including:

[0030] Input the user's initial voice data into a pre-set large language model;

[0031] The speaker's initial speech data is extracted by the network layer in the pre-set large language model to obtain the speaker's initial speech data.

[0032] Based on a preset number of text contents and the voiceprint features of the initial speech data, a preset number of cloned speech data are obtained.

[0033] Optionally, the method described above further includes:

[0034] Obtain the audio data to be converted;

[0035] The voice data to be converted is input into the singing voice conversion model to obtain the output converted voice data; where the converted voice data is the user's singing voice.

[0036] Secondly, this application provides a training device for a singing voice conversion model, comprising:

[0037] The acquisition unit is used to acquire the user's initial voice data; wherein the initial voice data represents the sound wave signal of a sentence of a preset length;

[0038] The cloning unit is used to obtain a preset number of cloned voice data based on the user's initial voice data and a preset large language model; wherein, the cloned voice data is voice data similar to the initial voice data;

[0039] The training unit is used to train the initial model based on a preset number of cloned speech data to obtain a trained singing conversion model; wherein, the singing conversion model is used to convert the singing of preset content into the user's singing.

[0040] Thirdly, this application provides an electronic device, including: a memory and a processor;

[0041] The memory stores instructions that the computer executes;

[0042] The processor executes computer execution instructions stored in memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.

[0043] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible embodiments of the first aspect.

[0044] Fifthly, this application provides a computer program product, comprising: a computer program that, when executed by a processor, implements the first aspect and / or various possible implementations of the first aspect.

[0045] The singing voice conversion model training method, apparatus, device, medium, and product provided in this application acquire initial user speech data, further obtain a preset number of cloned speech data based on the initial user speech data and a preset large language model, and then train the initial model based on the preset number of cloned speech data to obtain a trained singing voice conversion model. The initial speech data represents the sound wave signal of a sentence of preset length, and the cloned speech data are speech data similar to the initial speech data. The singing voice conversion model is used to convert the singing voice of preset content into the user's singing voice. By obtaining a preset number of cloned speech data from the user's initial speech data based on a preset large language model, the cost of acquiring a large amount of user speech data can be eliminated, thus enabling the training of the singing voice conversion model. The training method for the singing voice conversion model provided in this application saves the cost of training the singing voice conversion model and improves the accuracy of the model's singing voice conversion. Attached Figure Description

[0046] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0047] Figure 1 A flowchart illustrating a training method for a singing voice conversion model provided in this application;

[0048] Figure 2A flowchart illustrating the training method for another singing voice conversion model provided in this application;

[0049] Figure 3 A flowchart illustrating the training method for another singing voice conversion model provided in this application;

[0050] Figure 4 A schematic diagram of the framework of a singing voice conversion model provided in this application;

[0051] Figure 5 A schematic diagram of the structure of a training device for a singing voice conversion model provided in this application;

[0052] Figure 6 A schematic diagram of the structure of a training device for another singing voice conversion model provided in this application;

[0053] Figure 7 A schematic diagram of the structure of the electronic device provided in this application.

[0054] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0055] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0056] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with relevant laws, regulations and standards, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0057] The VITS (Variational Inference with adversarial learning for end-to-end Text-to-Speech) model is a highly expressive speech synthesis model that combines variational inference, normalizing flows, and adversarial training. In essence, text information is input into a trained VITS model, which outputs synthesized speech data.

[0058] The So-VITS-SVC (SoftVC VITS Singing Voice Conversion) model is a singing voice conversion model based on the VITS model. It can be used to convert one person's singing voice into another person's singing voice. For example, if user A's singing voice is input into a trained singing voice conversion model, user B's singing voice can be output.

[0059] This requires training the singing conversion model with a large amount of high-quality voice data (also known as singing data) from user B so that when user A's singing is input into the singing conversion model, user B's singing can be accurately output.

[0060] For example, if user B has poor singing ability, they may wish to convert some publicly licensed songs into their own singing voice for user B's personal enjoyment, or to carry out non-profit operations that comply with laws and regulations.

[0061] However, acquiring a large amount of high-quality voice data from user B is time-consuming and laborious, thus leading to the problem of high sampling costs in training the singing conversion model.

[0062] Therefore, training a singing voice conversion model faces the problem of high training costs or high data processing costs.

[0063] The singing voice conversion model training method, apparatus, device, medium, and product provided in this application acquire initial user speech data, further obtain a preset number of cloned speech data based on the initial user speech data and a preset large language model, and further train the initial model based on the preset number of cloned speech data to obtain a trained singing voice conversion model. Here, the initial speech data represents the sound wave signal of a sentence of preset length, the cloned speech data is speech data similar to the initial speech data, and the singing voice conversion model is used to convert the singing voice of preset content into the user's singing voice. By obtaining a preset number of cloned speech data from the user's initial speech data based on a preset large language model, the training of the singing voice conversion model can be achieved without incurring high costs in acquiring a large amount of user speech data.

[0064] Simultaneously, during the training of the initial model to obtain the trained singing voice conversion model, feature extraction is performed on each clone of speech data from a predetermined number of cloned speech data sets to obtain feature information for each clone. Then, based on the feature information of the predetermined number of cloned speech data sets and predetermined noise features, the initial model is trained to obtain the trained singing voice conversion model. The feature information includes at least one of the following: music theory features, frequency domain features, volume features, text features, audio autoencoder features, and voiceprint features. By using multiple feature information sets and predetermined noise features to train the initial model, the trained singing voice conversion model can improve the naturalness and accuracy of the user's singing voice output by the singing voice conversion model.

[0065] The training method for the singing voice conversion model provided in this application saves the cost of training the singing voice conversion model and improves the accuracy of the model's singing voice conversion.

[0066] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0067] Figure 1 This application provides a flowchart illustrating a training method for a singing voice conversion model. The execution entity of this method can be a server, host, or other device, such as... Figure 1 As shown, the method may include:

[0068] S101. Obtain the user's initial voice data; wherein, the initial voice data represents the sound wave signal of a sentence of a preset length.

[0069] The initial speech data can refer to the sound wave signal of a sentence of preset length. It can be understood that the speech content of the initial speech data can be a sentence of preset length, and the preset length can be a pre-set length, such as a character length of 6. For example, the text content of the sentence of preset length can be "Unity is strength".

[0070] The user's initial voice data can be represented as the initial voice data from the user, which can be understood as a sentence spoken by the user. The length of the sentence is within a preset length, and the form of the sentence is a sound wave signal.

[0071] Sound wave signals can refer to physical fluctuations generated by sound. These physical fluctuations can propagate through air or other media and are eventually captured by microphones or other sensors. For example, sound wave signals can be represented in the form of a waveform graph, where the horizontal axis represents time and the vertical axis represents amplitude. Amplitude can be understood as the intensity or volume of the sound wave. As the amplitude changes on the time axis, the sound wave signal can reflect the frequency and rhythm of the sound.

[0072] In one possible implementation, the user's initial voice data can be acquired via a microphone or other sensors, or the user can upload the recorded voice data to a relevant application, from which relevant personnel can retrieve the voice data.

[0073] In one possible implementation, the user's initial voice data can be stored in a preset storage space so that a preset number of cloned voice data can be obtained based on the user's initial voice data and a preset large language model.

[0074] It should be noted that the acquisition and storage of users' initial voice data have been done with the users' explicit consent and in accordance with relevant protection regulations.

[0075] It is understandable that the initial voice data obtained from users at this time has a small sample size and is easy to obtain. It does not require users to spend a lot of time or costs, and it can still achieve the subsequent training of the singing conversion model.

[0076] In one possible implementation, the singing voice conversion model of this application is a sub-model in a preset pipeline. The preset pipeline may include multiple sub-models. It can be understood that the relationship between the users providing initial speech data and the singing voice conversion models can be a one-to-one correspondence. That is, for a sub-model, the sub-model can be trained by a specified user. The trained sub-model can achieve the effect of converting the speech data to be converted (arbitrary input singing voice) into the user's singing voice.

[0077] S102. Based on the user's initial voice data and a preset large language model, obtain a preset number of cloned voice data; wherein, the cloned voice data is voice data similar to the initial voice data.

[0078] The pre-set large language model can refer to a pre-configured neural network model. In this application, the large speech model is used to output speech data similar to the user's initial speech data, i.e., cloned speech data, based on the user's initial speech data. This embodiment does not specifically limit the model structure of the large language model.

[0079] It is understandable that the similarity between the initial voice data and the cloned voice data can be reflected in the user's voice characteristics, such as the timbre, pitch, and speech rate of the initial voice data and the cloned voice data being similar.

[0080] The preset quantity refers to a pre-set number, which can be understood as the number of sentences in the text content of the cloned voice data. For example, the preset quantity is 100, the cloned voice data includes 100 sentences, and the voice duration of 100 sentences can be 5 minutes.

[0081] For example, the implementation logic of the preset large language model can be as follows: extract the sound features of the user's initial speech data; based on a preset number of text sequences, combined with the sound features of the user's initial speech data, use text-to-speech (TTS) technology to synthesize a preset number of cloned speech data, wherein the text content of the preset number of cloned speech data is a preset number of text sequences.

[0082] In one possible implementation, the preset large language model can be the speech synthesis model CosyVoice, or other large language models or large speech models. There are no restrictions here, as long as a preset number of cloned speech data can be obtained.

[0083] S103. Train the initial model based on a preset number of cloned voice data to obtain a trained singing voice conversion model; wherein, the singing voice conversion model is used to convert the singing voice of the preset content into the user's singing voice.

[0084] The initial model can be an untrained model. Through training, the internal parameters of the model can be adjusted so that it can be used to convert the singing of preset content into the singing of a specified user.

[0085] In one possible implementation, the initial model can be a model obtained by initially training an unoptimized model. This initial training can be performed using an open-source dataset of multiple singers' voices, rather than a pre-defined number of cloned voice data, to train the unoptimized model and obtain the initial model.

[0086] In one possible implementation, the framework of the initial model can be a framework that conforms to the So-VITS-SVC model.

[0087] It is understandable that by training an initial model with a preset number of cloned speech data, the similarity between the output singing voice of the initial model and the preset number of cloned speech data can be determined. If the similarity is greater than a preset similarity threshold, a trained singing voice conversion model can be obtained. For example, the similarity can be calculated using a loss function.

[0088] If the similarity is not greater than the preset similarity threshold, the model continues to be trained, and the internal parameters of the model are adjusted until the similarity is greater than the preset similarity threshold, thus obtaining the trained singing conversion model.

[0089] In one alternative implementation, step S103 may include:

[0090] Feature extraction is performed on each cloned speech data in a preset number of cloned speech data to obtain feature information for each cloned speech data; wherein, the feature information includes at least one of music theory features, frequency domain features, volume features, text features, audio autoencoder features, and voiceprint features; based on the feature information of the preset number of cloned speech data and preset noise features, the initial model is trained to obtain the trained singing conversion model.

[0091] In this process, feature extraction is performed on each cloned speech data within a predetermined number of cloned speech data sets to obtain the feature information of each cloned speech data set. It can be understood that feature extraction allows the acquisition of feature information from the cloned speech data sets.

[0092] The feature information of the cloned speech data may include at least one of the following features: music theory features, frequency domain features, volume features, text features, audio autoencoder features, and voiceprint features.

[0093] Music theory features can refer to features that include music-related attributes such as pitch, rhythm, and harmony of cloned speech data. For example, music theory features can be the fundamental frequency f0, the frequency of high notes, etc.

[0094] Frequency domain features can refer to the information of converting the acoustic signal of cloned speech data from the time domain to the frequency domain. For example, frequency domain features can be the Mel spectrum, etc.

[0095] Volume features can refer to the intensity or loudness of the sound wave signal in cloned speech data. For example, volume features can be energy features, etc., and volume features are obtained by calculating the energy or the sum of squares of the amplitude of the sound wave signal.

[0096] Text features can refer to the text content contained in cloned speech data. For example, text content can be extracted through speech recognition technology.

[0097] Audio autoencoder features refer to high-level features extracted using deep learning techniques such as autoencoders, which can automatically learn and capture complex patterns in the acoustic signals of speech data.

[0098] Voiceprint features refer to the unique vocal characteristics of a speaker, used to identify and distinguish different speakers. In essence, a user's vocal features can be extracted from cloned voice data, allowing the voice to be identified as that user's.

[0099] In one possible implementation, the feature information of cloned speech data may also include other features such as emotional features.

[0100] The advantage of this setup is that it makes the feature information of the cloned speech data as diverse as possible, thereby improving the accuracy of the trained singing conversion model and increasing the precision of the model's singing conversion.

[0101] In one optional implementation, feature extraction processing is performed on each cloned speech data in a preset number of cloned speech data to obtain feature information for each cloned speech data, which may include:

[0102] Each cloned speech data in a preset number of cloned speech data is preprocessed to obtain target speech data; wherein, the target speech data represents the preprocessed cloned speech data, and the preprocessing includes noise reduction processing and / or speech restoration processing; feature extraction processing is performed on the target speech data to obtain the feature information of the cloned speech data.

[0103] The purpose of preprocessing is to improve the quality of each cloned speech data in a predetermined number of cloned speech data, making it more suitable for subsequent feature extraction and model training.

[0104] Preprocessing may include noise reduction and / or speech restoration. Noise reduction refers to removing or reducing background noise in the cloned speech data, while speech restoration refers to repairing damaged or incomplete speech signals in the cloned speech data. By preprocessing each cloned speech data in a predetermined number of cloned speech data, the clarity and intelligibility of the cloned speech data can be improved.

[0105] In one possible implementation, preprocessing may also include silence trimming, pre-emphasis processing, etc. Silence trimming may refer to removing silent parts from the cloned speech data to reduce unnecessary data processing; pre-emphasis processing may refer to enhancing the high-frequency components in the cloned speech data through a high-pass filter to improve the clarity of the cloned speech data.

[0106] In one optional implementation, the feature information includes music theory features; feature extraction processing of the target speech data to obtain the feature information of the cloned speech data may include:

[0107] Based on a preset low-pass filter, the target speech data is filtered to obtain the music theory features of the cloned speech data.

[0108] Among them, the low-pass filter can be used to remove high-frequency noise or interference from cloned speech data.

[0109] Music theory features involve musical attributes in cloned speech data, such as pitch, rhythm, and harmony. Since these musical attributes are related to the basic structure and melody of music, they are usually concentrated in the lower frequency range. Music theory features of cloned speech data can be obtained by filtering the target speech data using a low-pass filter.

[0110] For example, music theory features can be the fundamental frequency f0, the frequency of treble notes, etc.

[0111] The advantage of this setup is that by extracting the music theory features of the cloned speech data, it can help the singing conversion model better understand the musical attributes in the cloned speech data, making the output singing of the trained singing conversion model more vivid and expressive.

[0112] In one optional implementation, the feature information includes frequency domain features; performing feature extraction processing on the target speech data to obtain the feature information of the cloned speech data may include:

[0113] Based on a preset Mel filter, the target speech data is processed by Mel spectrum filtering to obtain the frequency domain features of the cloned speech data.

[0114] In this context, Mel spectrum can refer to the Mel frequency diagram, frequency domain characteristics can refer to the Mel spectrum, Mel filter can refer to a filter distributed on the Mel frequency scale, and the Mel frequency scale can refer to a frequency scale that simulates human auditory perception. The Mel frequency scale has high resolution in the low-frequency range and low resolution in the high-frequency range, reflecting the sensitivity of the human ear to different frequencies.

[0115] The advantage of this setup is that by extracting the frequency domain features of the cloned speech data, the singing conversion model can take into account the sensitivity of the human ear to different frequencies, thereby improving the naturalness and accuracy of the singing output by the singing conversion model.

[0116] In one optional implementation, the feature information includes volume features; performing feature extraction processing on the target speech data to obtain the feature information of the cloned speech data may include:

[0117] The target speech data is segmented to obtain multiple frame signals; each frame signal is a part of the target speech data and each frame signal has the same length; the sum of squared amplitudes of each frame signal is determined and the sums of squared amplitudes of each frame signal are added together to obtain the volume characteristics of the cloned speech data; the sum of squared amplitudes represents the power spectral density.

[0118] The frame signal is a part of the target speech data. Each frame signal has the same length. For example, if the duration of the target speech data is 5 minutes, the target speech data can be segmented to obtain 300 frame signals, and the duration of each frame signal can be 1 second.

[0119] For each frame of signal, its sum of squared amplitudes is calculated, that is, the sum of the squares of the amplitude (i.e., signal intensity) of each sample point in the frame signal. It can be understood that the sum of squared amplitudes can also be called power spectral density (PSD), which reflects the energy distribution or volume distribution of the signal in that frame.

[0120] By summing the squared amplitudes of each frame of the signal, the volume characteristics of the cloned speech data can be obtained, that is, the energy distribution or volume distribution of the signal in the acoustic wave signal of the cloned speech data.

[0121] The advantage of this setup is that by extracting the volume features of the cloned speech data, the singing conversion model can take into account the energy distribution or volume distribution of the cloned speech data, and can better identify the emotional state and other aspects of the cloned speech data.

[0122] In one optional implementation, the feature information includes at least one of text features, audio autoencoder features, and voiceprint features; feature extraction processing of the target speech data to obtain the feature information of the cloned speech data may include:

[0123] Based on a pre-defined feature extraction model, feature extraction processing is performed on the target speech data to obtain the feature information of the cloned speech data.

[0124] The preset feature extraction model can be at least one of the following: HuBERT audio autoencoder model, Whisper speech recognition model, Contextvec speech representation learning model, and voiceprint model.

[0125] The HuBERT model can refer to a self-supervised learning model. It is based on the BERT (Bidirectional Encoder Representations from Transformers) architecture but with modifications to adapt it to speech sequences. Using the HuBERT model, HuBERT features can be extracted from cloned speech data. These HuBERT features can represent the autoencoder features of clustered audio data.

[0126] The Whisper model can refer to a Transformer-based encoder-decoder model. It can be used to extract textual features from cloned speech data, known as Whisper features. Its implementation logic involves mapping the audio spectrum of the target speech data to tokens, and then converting the tokens back into text.

[0127] The Contextvec model refers to a speech representation learning framework that uses a student network to learn audio representations, thereby achieving state-of-the-art (SOTA) performance in tasks such as voice verification and speech understanding. The Contextvec model allows for the extraction of Contextvec features from cloned speech data, which characterize audio features.

[0128] A voiceprint model can refer to a pre-defined model that can be used to extract voiceprint features. Voiceprint features can also be understood as vector representations of speaker characteristics, and can be represented by X-vector features. In one possible implementation, voiceprint features may include, but are not limited to, a set of Mel-frequency cepstral coefficients (MFCC), linear prediction coefficients (LPC), and perceptual linear prediction (PLP).

[0129] The advantage of this setup is that by extracting HuBERT features, Whisper features, and Contextvec features from the cloned speech data, the singing conversion model can take into account important factors such as audio and text in the cloned speech data. By extracting X-vector features from the cloned speech data, the singing conversion model can take into account the speaker's representation in the cloned speech data, so that the output singing voice is as similar as possible to the user's singing voice.

[0130] In one optional implementation, an initial model is trained based on feature information from a preset number of cloned speech data and preset noise features to obtain a trained singing conversion model, which may include:

[0131] The feature information of each cloned speech data in a preset number of cloned speech data is fused with preset noise features to obtain target features; wherein, the target features represent the feature information of the cloned speech data and the preset noise features; the pre-collected speech data to be trained is acquired, and the initial model is trained based on the speech data to be trained, the target features, and the cloned speech data to obtain the trained singing conversion model; wherein, the speech data to be trained is non-user speech data.

[0132] The preset noise feature can refer to a pre-set noise feature that can characterize Gaussian white noise. In one possible implementation, the dimension of the noise feature is the same as the dimension of the feature information of each cloned speech data in the preset number of cloned speech data. For example, the dimension of the noise feature and the dimension of the feature information of each cloned speech data in the preset number of cloned speech data are both 192.

[0133] Feature fusion processing can refer to the summation of the feature information of each cloned speech data in a preset number of cloned speech data with preset noise features.

[0134] For example, for each feature in the feature information of each cloned speech data in a preset number of cloned speech data, summation is performed to obtain feature information of a certain dimension, such as 192 dimensions.

[0135] Furthermore, the feature information of each cloned speech data in the preset number of cloned speech data is summed with the preset noise features, wherein the dimension of the noise features can also be the aforementioned 192 dimensions.

[0136] By extracting the feature information of each cloned speech data from a preset number of cloned speech data, and then fusing the feature information of each cloned speech data with preset noise features to obtain target features, instead of directly adding noise to the cloned speech data, the interference of feature extraction on each cloned speech data in the preset number of cloned speech data can be avoided. At the same time, the addition of noise features to the target features enables the singing conversion model to have better naturalness, and its output singing is closer to the naturalness of human voice.

[0137] The training voice data can refer to non-user voice data. In one possible implementation, the training voice data can be open-source voice data from multiple people.

[0138] For example, training speech data can be obtained from an open-source platform. It should be noted that obtaining training speech data from an open-source platform is authorized by the platform and complies with relevant laws and regulations.

[0139] It is understandable that by training the initial model based on the speech data to be trained, the target features, and the cloned speech data, a trained singing conversion model can be obtained.

[0140] In one optional implementation, the initial model is trained based on the speech data to be trained, target features, and cloned speech data to obtain a trained singing conversion model, which may include:

[0141] The training speech data and target features are input into the initial model, and the predicted singing information is output. If the similarity between the predicted singing information and the cloned speech data meets the preset threshold, the trained singing conversion model is obtained.

[0142] The process involves inputting the speech data to be trained and the target features into the initial model, which then outputs the predicted singing information.

[0143] The predicted singing information can be the predicted singing data signal. The text features in the singing data signal are derived from the speech data to be trained, and the timbre-related features in the singing data signal are derived from the target features.

[0144] If the similarity between the predicted singing information and the cloned speech data meets a preset threshold, then the trained singing conversion model can be obtained.

[0145] For example, similarity can be calculated using a loss function. The result of the loss function can be a scalar value.

[0146] In one possible implementation, the similarity between the predicted singing voice information and the cloned speech data can be obtained by adding the similarities between the predicted singing voice information and the cloned speech data at least one of the following features: music theory features, frequency domain features, volume features, text features, audio autoencoder features, and voiceprint features.

[0147] The preset threshold can refer to a preset threshold used to measure the similarity between the predicted singing information and the cloned voice data. For example, the preset threshold can be set to 0.9.

[0148] If the similarity between the predicted singing voice information and the cloned speech data is, for example, 0.95, which meets the preset threshold, that is, the similarity between the predicted singing voice information and the cloned speech data is 0.95, which is greater than the preset threshold of 0.9, then the trained singing voice conversion model can be obtained.

[0149] This application provides a training method for a singing voice conversion model. By using the user's initial speech data and a pre-defined large language model, a predetermined number of cloned speech data can be obtained, eliminating the need for costly acquisition of large amounts of user speech data to train the singing voice conversion model. Simultaneously, the initial model is trained based on the feature information of the predetermined number of cloned speech data and pre-defined noise features, resulting in a trained singing voice conversion model. The feature information includes at least one of music theory features, frequency domain features, volume features, text features, audio autoencoder features, and voiceprint features. By using multiple feature information and pre-defined noise features to train the initial model, the trained singing voice conversion model can be obtained, improving the naturalness and accuracy of the user's singing voice output. The training method for the singing voice conversion model provided in this application saves the cost of training the model and improves the accuracy of the singing voice conversion.

[0150] Figure 2 The flowchart illustrates another training method for a singing voice conversion model provided in this application. The execution entity of this method can be a server, host, or other device, such as... Figure 2 As shown, the method may include:

[0151] S201. Obtain the user's initial voice data; wherein, the initial voice data represents the sound wave signal of a sentence of a preset length.

[0152] For example, this step can refer to step S101 above, and will not be repeated here.

[0153] S202. Input the user's initial voice data into the preset large language model.

[0154] In one possible implementation, the pre-defined large language model can be the CosyVoice speech synthesis model, which may include a voiceprint extraction module and a text encoder module.

[0155] S203. Through the network layer in the preset large language model, the user's initial speech data is extracted to obtain the voiceprint features of the initial speech data.

[0156] In one possible implementation, the network layers of the pre-defined large language model may include the aforementioned speaker extraction model. The speaker extraction module can extract speaker features from the user's initial speech data to obtain the speaker features of the initial speech data. The speaker features can be represented using X-vector features.

[0157] In one possible implementation, the voiceprint features may include, but are not limited to, a set of Mel frequency cepstral coefficients (MFCC), linear prediction coefficients (LPC), and perceptual linear prediction (PLP).

[0158] S204. Based on the preset number of text contents and the voiceprint features of the initial speech data, obtain a preset number of cloned speech data.

[0159] In one possible implementation, the pre-defined large language model also includes the aforementioned text encoder module, which can be used to input a preset number of text contents and output a preset number of text encoded sequences.

[0160] In one possible implementation, the pre-defined large language model also includes a Text-to-token LM module, a flow-matching module, and a vocoder module.

[0161] It is understandable that, based on a preset number of text encoding sequences and the voiceprint features of the initial speech data as input to the Text-to-token LM module, a predicted semantic token can be output. Then, an acoustic token is generated through the flow-matching module. Finally, the generated acoustic token is used by the vocoder module to generate a sound wave signal, which is the preset number of cloned speech data.

[0162] S205. Based on a preset number of cloned speech data, train the initial model to obtain a trained singing voice conversion model; wherein, the singing voice conversion model is used to convert the singing voice of the preset content into the user's singing voice.

[0163] For example, this step can refer to step S103 above, and will not be repeated here.

[0164] This application provides an alternative training method for a singing voice conversion model. Using a pre-defined large language model, and based on a pre-defined amount of text content and the voiceprint features of initial speech data, a pre-defined number of cloned speech data are obtained. This method preserves the user's voiceprint features and can quickly and efficiently clone a pre-defined number of speech data without incurring the high cost of acquiring large amounts of user speech data. Therefore, this alternative training method for a singing voice conversion model saves on the cost of training such a model.

[0165] Figure 3 The flowchart illustrates another training method for the singing voice conversion model provided in this application. The execution entity of this method can be a server, host, or other device, such as... Figure 3 As shown, the method may include:

[0166] S301. Obtain the voice data to be converted.

[0167] The voice data to be converted can be non-user voice data.

[0168] For example, the voice data to be converted can be non-user singing data.

[0169] S302. Input the speech data to be converted into the singing conversion model to obtain the output converted speech data; wherein, the converted speech data is the user's singing voice.

[0170] It is understandable that the trained singing conversion model has learned how to convert any input singing voice into the singing voice of a specified user. By inputting non-user singing voice data into the singing conversion model, at least one feature from the music theory features, frequency domain features, volume features, text features, audio autoencoder features, and voiceprint features of the non-user singing voice data is extracted as the feature to be adjusted. After processing by the network layer in the singing conversion model, the feature to be adjusted can be adjusted to be similar to the feature information of the user's cloned speech data, which is the adjusted feature.

[0171] Using the vocoder in the singing conversion model, the adjusted features are converted into sound wave signals, i.e., converted speech data, which can be understood as the user's singing voice.

[0172] The alternative training method for a singing voice conversion model provided in this application allows for training of the model using only the sound wave signal of a user's pre-defined phrase, without the need for costly acquisition of large amounts of user speech data. When the trained singing voice conversion model is used, it can still accurately convert the speech data to be converted (arbitrarily input singing voice) into the singing voice of a specified user. This alternative training method saves the cost of training the singing voice conversion model and improves the accuracy of the model's singing voice conversion.

[0173] Figure 4 This application provides a schematic diagram of the framework of a singing voice conversion model, as shown below. Figure 4 As shown, the framework of the singing conversion model includes a prior encoder module, a posterior encoder module, and a stream module.

[0174] For example, the singing voice conversion model in this embodiment can be the So-VITS-SVC model.

[0175] The role of the prior encoder module in the So-VITS-SVC model is to process the input audio features and generate a prior distribution.

[0176] Compared to the prior encoder of the traditional VITS model, the input of the prior encoder of the So-VITS-SVC model is more complex. The input of the prior encoder can include at least three features: music theory features (fundamental frequency f0), Whisper features, and HuBERT features.

[0177] In addition, for example, the input of the prior encoder of the So-VITS-SVC model may also include at least one or more of the following features: frequency domain features, volume features, and Whisper features, to increase the richness of the input features.

[0178] In one possible implementation, after obtaining the music theory features (fundamental frequency f0), Whisper features, and HuBERT features, random jitter is added, or noise features corresponding to preset Gaussian white noise are used to ensure the naturalness of the audio. These four features are all processed into size-192 vectors, and then summed as the input to the prior encoder of the So-VITS-SVC model.

[0179] In one possible implementation, the prior encoder may also include a speaker predictor. The speaker predictor aims to maximize the difference between the predicted speaker and the real speaker, that is, to eliminate the influence of speaker information on the input features. Therefore, the speaker predictor will incorporate gradient inversion.

[0180] The role of the posterior encoder in the so-VITS-SVC model is to generate the posterior distribution. The speaker information input into the posterior encoder is not a simple ID, but voiceprint features.

[0181] It is understandable that the posterior encoder is responsible for encoding the input voiceprint features, and the decoding process can be a process of continuously upsampling from the latent variable Z until the speech waveform is recovered, so that the so-VITS-SVC model can generate a singing voice similar to that of the user.

[0182] The role of the stream module in the So-VITS-SVC model can be to be responsible for generating the inference process of the singing conversion model.

[0183] As can be understood, the streaming module gradually transforms source audio features into target audio features through a series of reversible transformations. These transformations can be linear or nonlinear; they work together on the input audio features, simulating the statistical properties of the audio signal, and ultimately generating a sound similar to that of the user.

[0184] For example, the flow module increases the expressive power of acoustic signals by setting a transformation function f that transforms the posterior distribution q to the prior distribution p.

[0185] This application provides a framework for a singing voice conversion model, including a prior encoder module, a posterior encoder module, and a stream module, which work together to convert the voice data to be converted (arbitrary input singing voice) into the singing voice of a specified user, thereby improving the accuracy of the singing voice conversion model.

[0186] Figure 5 A schematic diagram of the structure of a training device for a singing voice conversion model provided in this application is shown below. Figure 5 As shown, the training device 50 for the singing conversion model includes: an acquisition unit 501, a cloning unit 502, and a training unit 503.

[0187] The acquisition unit 501 is used to acquire the user's initial voice data; wherein the initial voice data represents the sound wave signal of a sentence of a preset length;

[0188] The cloning unit 502 is used to obtain a preset number of cloned voice data based on the user's initial voice data and a preset large language model; wherein, the cloned voice data is voice data similar to the initial voice data.

[0189] Training unit 503 is used to train the initial model based on a preset number of cloned speech data to obtain a trained singing conversion model; wherein, the singing conversion model is used to convert the singing of preset content into the user's singing.

[0190] Figure 6 A schematic diagram of the structure of a training device for another singing voice conversion model provided in this application is shown below. Figure 6 As shown, the training device 60 for the singing conversion model includes: an acquisition unit 601, a cloning unit 602, and a training unit 603. The training unit 603 further includes a feature extraction module 6031 and a training module 6032.

[0191] In an optional example, the feature extraction module 6031 is used to perform feature extraction processing on each cloned speech data in a preset number of cloned speech data to obtain feature information of each cloned speech data; wherein, the feature information includes at least one of music theory features, frequency domain features, volume features, text features, audio autoencoder features, and voiceprint features.

[0192] In an optional example, the training module 6032 is used to train the initial model based on the feature information of a preset number of cloned speech data and preset noise features to obtain a trained singing conversion model.

[0193] In an optional example, the feature extraction module 6031 may also include a first extraction submodule and a second extraction submodule.

[0194] The first extraction submodule is used to preprocess each cloned speech data in a preset number of cloned speech data to obtain target speech data; wherein, the target speech data represents the preprocessed cloned speech data, and the preprocessing includes noise reduction processing and / or speech restoration processing.

[0195] The second extraction submodule is used to perform feature extraction processing on the target speech data to obtain the feature information of the cloned speech data.

[0196] In an optional example, the feature information includes music theory features. The second extraction submodule can also be specifically used to filter the target speech data based on a preset low-pass filter to obtain the music theory features of the cloned speech data.

[0197] In an optional example, the feature information includes frequency domain features. The second extraction submodule can also be specifically used to perform Mel spectrum filtering on the target speech data based on a preset Mel filter to obtain the frequency domain features of the cloned speech data.

[0198] In an optional example, the feature information includes volume features. The second extraction submodule can also be specifically used to segment the target speech data to obtain multiple frame signals. The frame signals are a part of the target speech data, and each frame signal has the same length. The sum of squared amplitudes of each frame signal is determined, and the sums of squared amplitudes of each frame signal are added together to obtain the volume features of the cloned speech data. The sum of squared amplitudes represents the power spectral density.

[0199] In an optional example, training module 6032 also includes a first training submodule and a second training submodule.

[0200] The first training submodule is used to fuse the feature information of each cloned speech data in a preset number of cloned speech data with preset noise features to obtain target features; wherein, the target features represent the feature information of the cloned speech data and the preset noise features.

[0201] The second training submodule is used to acquire pre-collected speech data to be trained, and to train the initial model based on the speech data to be trained, target features, and cloned speech data to obtain a trained singing conversion model; wherein, the speech data to be trained is non-user speech data.

[0202] In an optional example, the second training submodule can also be specifically used to input the speech data to be trained and the target features into the initial model and output the predicted singing information.

[0203] If the similarity between the predicted singing information and the cloned speech data meets a preset threshold, then the trained singing conversion model is obtained.

[0204] In an optional example, cloning unit 602 can also be specifically used to input the user's initial speech data into a preset large language model;

[0205] The speaker's initial speech data is extracted by the network layer in the pre-set large language model to obtain the speaker's initial speech data.

[0206] Based on a preset number of text contents and the voiceprint features of the initial speech data, a preset number of cloned speech data are obtained.

[0207] In an optional example, the training device 60 for the singing conversion model also includes an application unit.

[0208] The application unit is used to acquire the speech data to be converted;

[0209] The voice data to be converted is input into the singing voice conversion model to obtain the output converted voice data; where the converted voice data is the user's singing voice.

[0210] Figure 7 A schematic diagram of the structure of the electronic device provided in this application, such as... Figure 7 As shown, the electronic device 70 provided in this embodiment includes at least one processor 701 and a memory 702. Optionally, the device 70 further includes a communication component 703. The processor 701, memory 702, and communication component 703 are connected via a bus 704.

[0211] In a specific implementation, at least one processor 701 executes computer execution instructions stored in memory 702, causing at least one processor 701 to perform the above-described method.

[0212] The specific implementation process of processor 701 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0213] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0214] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0215] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0216] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0217] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0218] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0219] It should be further noted that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0220] It should be understood that the above-described device embodiments are merely illustrative, and the device of this application can also be implemented in other ways. For example, the division of units / modules in the above embodiments is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units, modules, or components may be combined, or integrated into another system, or some features may be ignored or not executed.

[0221] Furthermore, unless otherwise specified, the functional units / modules in the various embodiments of this application can be integrated into one unit / module, or each unit / module can exist physically separately, or two or more units / modules can be integrated together. The integrated units / modules described above can be implemented in hardware or as software program modules.

[0222] When integrated units / modules are implemented in hardware, the hardware can be digital circuits, analog circuits, etc. The physical implementation of the hardware structure includes, but is not limited to, transistors, memristors, etc. Unless otherwise specified, the processor can be any suitable hardware processor, such as a CPU, GPU, FPGA, DSP, and ASIC, etc. Unless otherwise specified, the storage unit can be any suitable magnetic or magneto-optical storage medium, such as Resistive Random Access Memory (RRAM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), Enhanced Dynamic Random Access Memory (EDRAM), High-Bandwidth Memory (HBM), Hybrid Memory Cube (HMC), etc.

[0223] If the integrated unit / module is implemented as a software program module and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0224] In the above embodiments, the descriptions of each embodiment have their own emphasis. Parts not described in detail in a particular embodiment can be referred to in the relevant descriptions of other embodiments. The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as these combinations of technical features do not contradict each other, they should be considered within the scope of this specification.

[0225] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0226] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A training method for a singing voice conversion model, characterized in that, include: Acquire the user's initial voice data; wherein the initial voice data represents the sound wave signal of a sentence of a preset length; Based on the user's initial voice data and a preset large language model, a preset number of cloned voice data are obtained; wherein, the cloned voice data is voice data similar to the initial voice data; Based on the preset number of cloned voice data, the initial model is trained to obtain a trained singing voice conversion model; wherein, the singing voice conversion model is used to convert the singing voice of the preset content into the user's singing voice.

2. The method according to claim 1, characterized in that, Based on the preset number of cloned speech data, the initial model is trained to obtain a trained singing conversion model, including: Feature extraction processing is performed on each cloned speech data in the preset number of cloned speech data to obtain feature information of each cloned speech data; wherein, the feature information includes at least one of music theory features, frequency domain features, volume features, text features, audio autoencoder features, and voiceprint features; Based on the feature information of the preset number of cloned speech data and the preset noise features, the initial model is trained to obtain a trained singing conversion model.

3. The method according to claim 2, characterized in that, Feature extraction processing is performed on each cloned speech data in the preset number of cloned speech data to obtain the feature information of each cloned speech data, including: Each cloned speech data in the preset number of cloned speech data is preprocessed to obtain target speech data; wherein, the target speech data represents the preprocessed cloned speech data, and the preprocessing includes noise reduction processing and / or speech restoration processing; Feature extraction processing is performed on the target speech data to obtain the feature information of the cloned speech data.

4. The method according to claim 3, characterized in that, The feature information includes music theory features; feature extraction processing is performed on the target speech data to obtain the feature information of the cloned speech data, including: Based on a preset low-pass filter, the target speech data is filtered to obtain the music theory features of the cloned speech data.

5. The method according to claim 3, characterized in that, The feature information includes frequency domain features; feature extraction processing is performed on the target speech data to obtain the feature information of the cloned speech data, including: Based on a preset Mel filter, the target speech data is subjected to Mel spectrum filtering processing to obtain the frequency domain features of the cloned speech data.

6. The method according to claim 3, characterized in that, The feature information includes volume features; feature extraction processing is performed on the target speech data to obtain the feature information of the cloned speech data, including: The target speech data is segmented to obtain multiple frame signals; wherein each frame signal is a part of the target speech data and each frame signal has the same length. The sum of squared amplitudes of each frame of signal is determined, and the sums of squared amplitudes of each frame of signal are added together to obtain the volume characteristics of the cloned speech data; wherein the sum of squared amplitudes represents the power spectral density.

7. The method according to claim 2, characterized in that, Based on the feature information of the preset number of cloned speech data and the preset noise features, the initial model is trained to obtain a trained singing conversion model, including: The feature information of each cloned speech data in the preset number of cloned speech data is fused with preset noise features to obtain target features; wherein, the target features represent the feature information of the cloned speech data and the preset noise features. Acquire pre-collected training voice data, and train the initial model based on the training voice data, the target features, and the cloned voice data to obtain a trained singing voice conversion model; wherein, the training voice data is voice data not belonging to the user.

8. The method according to claim 7, characterized in that, Based on the speech data to be trained, the target features, and the cloned speech data, the initial model is trained to obtain a trained singing conversion model, including: The speech data to be trained and the target features are input into the initial model, and the predicted singing information is output. If the similarity between the predicted singing information and the cloned speech data meets a preset threshold, then the trained singing conversion model is obtained.

9. The method according to claim 1, characterized in that, Based on the user's initial voice data and a preset large language model, a preset number of cloned voice data are obtained, including: The user's initial voice data is input into the preset large language model; The user's initial speech data is extracted by the network layer in the preset large language model to obtain the voiceprint features of the initial speech data. Based on a preset number of text contents and the voiceprint features of the initial voice data, a preset number of cloned voice data are obtained.

10. The method according to claim 1, characterized in that, Also includes: Obtain the audio data to be converted; The voice data to be converted is input into the singing conversion model to obtain the output converted voice data; wherein, the converted voice data is the user's singing voice.

11. A training device for a singing voice conversion model, characterized in that, include: An acquisition unit is used to acquire the user's initial voice data; wherein the initial voice data represents the sound wave signal of a sentence of a preset length; The cloning unit is used to obtain a preset number of cloned voice data based on the user's initial voice data and a preset large language model; wherein the cloned voice data is voice data similar to the initial voice data. The training unit is used to train the initial model based on the preset number of cloned speech data to obtain a trained singing voice conversion model; wherein the singing voice conversion model is used to convert the singing voice of the preset content into the singing voice of the user.

12. An electronic device, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1-10.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-10.

14. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method described in any one of claims 1-10.

Images