Song conversion model training method, song timbre conversion method and related products

Abstract

The invention relates to a song conversion model training method, a song timbre conversion method, computer equipment, a computer readable storage medium and a computer program product, relates to the technical field of song synthesis, and can improve the efficiency of song timbre conversion by a user. The singing conversion model training method comprises the steps of obtaining a sample song and a singer timbre description text paired with the sample song; obtaining tone characteristics of the sample song by an audio coding module in the singing conversion model, and obtaining text characteristics corresponding to the tone description text of the singer by a text coding module in the singing conversion model; according to the difference between the timbre features and the text features, adjusting model parameters of a singing sound conversion model to obtain a trained singing sound conversion model; the trained song conversion model is used for extracting text features corresponding to the input timbre description text through the text coding module, obtaining target timbre features according to the text features corresponding to the timbre description text, and performing song timbre conversion.

Description

Technical Field

[0001] The present application relates to the technical field of singing voice synthesis, and in particular to a singing voice conversion model training method, a song timbre conversion method, a computer device, a computer-readable storage medium and a computer program product. Background Art

[0002] With the development of computer technology, singing voice conversion technology has become more and more popular. Singing voice conversion is a process of converting singing voice in audio from one sound source to another target sound source while keeping the content and rhythm of the song unchanged.

[0003] In the related art, when performing singing voice conversion, characteristic information representing the timbre characteristics is obtained from the sample recording, and then the information is used to control the singing voice conversion process. However, although this method is effective, it requires the user to first search for or record a ready-made recording with a specified timbre, which reduces the efficiency of singing voice conversion. Summary of the invention

[0004] Based on this, it is necessary to provide a singing voice conversion model training method, a song timbre conversion method, a computer device, a computer-readable storage medium and a computer program product to address the above-mentioned technical problems.

[0005] In a first aspect, the present application provides a singing voice conversion model training method, comprising:

[0006] Obtaining a sample song and a voice description text of a singer paired with the sample song;

[0007] The audio encoding module in the singing voice conversion model obtains the timbre characteristics of the sample song, and the text encoding module in the singing voice conversion model obtains the text characteristics corresponding to the timbre description text of the singer;

[0008] According to the difference between the timbre features and the text features, the model parameters of the singing voice conversion model are adjusted to obtain a trained singing voice conversion model; the trained singing voice conversion model is used to extract the text features corresponding to the input timbre description text through the text encoding module, and obtain the target timbre features according to the text features corresponding to the timbre description text to perform song timbre conversion.

[0009] In one embodiment, adjusting the model parameters of the singing voice conversion model according to the difference between the timbre feature and the text feature to obtain a trained singing voice conversion model includes:

[0010] Acquire phoneme pronunciation feature distribution information of the sample song according to the text feature and the pronunciation information of the sample song;

[0011] Acquire audio feature distribution information used to decode the sample song, and perform distribution mapping on the phoneme pronunciation feature distribution information and the audio feature distribution information by a distribution mapping module in the singing voice conversion model to obtain a distribution mapping result;

[0012] According to the difference between the timbre feature and the text feature, and the difference between the distribution mapping result and the reference distribution mapping result, the model parameters of the singing voice conversion model are adjusted to obtain a trained singing voice conversion model.

[0013] In one embodiment, adjusting the model parameters of the singing voice conversion model according to the difference between the timbre feature and the text feature, and the difference between the distribution mapping result and the reference distribution mapping result, comprises:

[0014] The distribution decoding module in the singing voice conversion model decodes the corresponding predicted song according to the audio feature distribution information;

[0015] Determine a first loss value according to the difference between the timbre feature and the text feature, determine a second loss value according to the difference between the distribution mapping result and the reference distribution mapping result, and determine a third loss value according to the difference between the predicted song and the sample song;

[0016] A model loss value is determined according to the first loss value, the second loss value and the third loss value, and a model parameter of the singing voice conversion model is adjusted according to the model loss value.

[0017] In one embodiment, the distribution decoding module in the singing voice conversion model decodes the corresponding predicted song according to the audio feature distribution information, including:

[0018] The audio feature distribution information, the text features and the pitch information of the sample song are input into the distribution decoding module in the singing voice conversion model, and the predicted song is obtained according to the output result of the distribution decoding module.

[0019] In one of the embodiments, the reference distribution mapping result includes the phoneme pronunciation feature distribution information or the audio feature distribution information;

[0020] The difference between the distribution mapping result and the reference distribution mapping result is determined by the following steps:

[0021] If the distribution mapping result is an audio feature distribution mapping result obtained by the distribution mapping module performing forward mapping on the audio feature distribution information, determining a difference between the audio feature distribution mapping result and the phoneme pronunciation feature distribution information;

[0022] If the distribution mapping result is a phoneme feature distribution mapping result obtained by the distribution mapping module performing reverse mapping on the phoneme pronunciation feature distribution information, a difference between the phoneme feature distribution mapping result and the audio feature distribution information is determined.

[0023] In one of the embodiments, the pronunciation information of the sample song includes pitch information of the sample song and phoneme features of the sample song;

[0024] The step of obtaining the phoneme pronunciation feature distribution information of the sample song according to the text feature and the pronunciation information of the sample song includes:

[0025] Inputting the text features, the pitch information of the sample song and the phoneme features of the sample song into the text encoding module to obtain a phoneme prior distribution output by the text encoding module;

[0026] The phoneme prior distribution is used as the phoneme pronunciation feature distribution information of the sample song.

[0027] In a second aspect, the present application also provides a song timbre conversion method, comprising:

[0028] Obtaining a song to be converted and timbre configuration information including at least timbre description text;

[0029] Input the pronunciation information of the song to be converted and the timbre configuration information into a trained singing voice conversion model, and obtain the song after the timbre conversion is output by the singing voice conversion model;

[0030] Wherein, the singing voice conversion model is trained according to the singing voice conversion model training method as described in any of the above items.

[0031] In one of the embodiments, the timbre configuration information further includes material audio with a specified timbre;

[0032] The step of inputting the pronunciation information of the song to be converted and the timbre configuration information into a trained singing voice conversion model to obtain the song after the singing voice conversion model outputs the timbre conversion, comprising:

[0033] The text encoding module in the singing voice conversion model performs feature extraction on the timbre description text to obtain text features of the timbre description text, and the audio encoding module in the singing voice conversion model performs feature extraction on the material audio to obtain timbre features of the material audio;

[0034] According to the feature weights of the text features of the timbre description text and the timbre features of the material audio, feature fusion is performed on the text features of the timbre description text and the timbre features of the material audio to obtain a target timbre feature;

[0035] The song after timbre conversion is output according to the target timbre feature and the pronunciation information of the song to be converted.

[0036] In a third aspect, the present application also provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the steps of the singing voice conversion model training method as described in any one of the above items or the steps of the song timbre conversion method as described in any one of the above items.

[0037] In a fourth aspect, the present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the singing voice conversion model training method as described in any one of the above items or the steps of the song timbre conversion method as described in any one of the above items.

[0038] In a fifth aspect, the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the singing voice conversion model training method as described in any of the above items or the steps of the song timbre conversion method as described in any of the above items.

[0039] The above-mentioned singing voice conversion model training method, song timbre conversion method, computer device, computer-readable storage medium and computer program product, by using paired sample songs and singers' timbre descriptions to train the audio encoding module and the text encoding module in the singing voice conversion model, can make the text features output by the text encoding module and the timbre features output by the audio encoding module as close as possible, so that after the training is completed, the text features can be used to replace the timbre features. When timbre conversion is required, it is only necessary to input any timbre description text related to the target timbre, without the need for users to record in advance or find sample recordings, thereby effectively improving the singing voice conversion efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0041] Figure 1 A schematic flow chart of a singing voice conversion model training method in one embodiment;

[0042] Figure 2 A structural schematic diagram of a singing voice conversion model in one embodiment;

[0043] Figure 3A schematic diagram of a flow chart of a song timbre conversion method in one embodiment;

[0044] Figure 4a A schematic flow chart of another song timbre conversion method in one embodiment;

[0045] Figure 4b A schematic flow chart of another song timbre conversion method in one embodiment;

[0046] Figure 4c A schematic flow chart of another song timbre conversion method in one embodiment;

[0047] Figure 5 is an internal structure diagram of a computer device in one embodiment;

[0048] Figure 6 The figure is a diagram of the internal structure of another computer device in one embodiment. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0050] In order to enable those skilled in the art to better understand the present application, the relevant technologies are introduced below.

[0051] Singing voice conversion is a process of converting the singing sound in audio from one sound source to another target sound source, while keeping the content and rhythm of the song unchanged. In the related art, when performing singing voice conversion, feature information representing the timbre characteristics is obtained from the sample recording, and then the information is used to control the singing voice conversion process. However, although this method is effective, it requires the user to first find or record a ready-made recording with a specified timbre. In order to alleviate this problem, the model can be decoupled from the relationship between audio and singing style and singers, and multiple dimensions can be disassembled. However, this method requires complex design and meticulous model training, and the iteration and maintenance of the model require a large cost. It can be seen that the related art has the problem of low efficiency of singing voice conversion.

[0052] Based on this, the present application provides a singing voice conversion model training method, a song timbre conversion method, a computer device, a computer-readable storage medium and a computer program product to at least solve the above-mentioned technical problems.

[0053] In one embodiment, Figure 1As shown, a singing voice conversion model training method is provided. This embodiment uses the method applied to a terminal as an example for illustration. It can be understood that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0054] S101, obtaining a sample song and a voice description text of a singer paired with the sample song.

[0055] The sample song is a song containing singing voices, and the sample song may be a song sung by a real person, or a song obtained by singing voice synthesis technology. In some examples, the sample song may be recorded in the form of an audio file, such as a wav audio file, and the sample song may be dry voice data containing the voice of a singer, and the sample song may be obtained by removing noise from the audio of the song containing the voice of a singer and cutting out blank segments therein.

[0056] The singer's timbre description text may be text information used to describe the singer's timbre characteristics, such as "sweet child's voice", "mature baritone", etc.

[0057] In this step, multiple training samples for training the singing voice conversion model can be obtained, each training sample includes a sample song and a singer's timbre description text paired with the sample song, wherein the singer's timbre description text paired with the sample song refers to the text used to describe the timbre characteristics of the singer corresponding to the sample song.

[0058] S102, the audio encoding module in the singing voice conversion model obtains the timbre characteristics of the sample song, and the text encoding module in the singing voice conversion model obtains the text characteristics corresponding to the singer's timbre description text.

[0059] After obtaining the sample song and the paired singer's timbre description text, on the one hand, the sample song can be input into the audio encoding module in the singing voice conversion model, and the audio encoding module obtains the timbre features of the sample song. For example, the Mel spectrum corresponding to the sample song can be extracted and input into the Mel encoder of the audio encoding module, and the output is used as the timbre feature; on the other hand, the singer's timbre description text can be input into the text encoding module in the singing voice conversion model, and the text encoding module obtains the text features corresponding to the singer's timbre description text. In some examples, the singer's timbre description text can be preprocessed by a pre-trained language model, and the preprocessed information can be input into the text encoding module for feature extraction.

[0060] S103, according to the difference between the timbre features and the text features, the model parameters of the singing voice conversion model are adjusted to obtain a trained singing voice conversion model; the trained singing voice conversion model is used to extract the text features corresponding to the input timbre description text through the text encoding module, and obtain the target timbre features according to the text features corresponding to the timbre description text to perform song timbre conversion.

[0061] Specifically, the embodiment of the present application proposes to add a text-controlled timbre function to the singing voice conversion model, that is, to control the timbre used in the singing voice conversion process through the singer's timbre description text. To this end, after obtaining the timbre features and text features, the difference between the timbre features and the text features can be determined, and the model parameters of the singing voice conversion model can be adjusted according to the difference, so that for the same sample song, the timbre features output by the audio encoding module and the text features output by the text encoding module of the singing voice conversion model can be as close as possible, so that the text features corresponding to the singer's timbre description text can be used as the singer's timbre features, instead of the timbre features extracted from the sample song itself.

[0062] In this regard, the model parameters of the singing voice conversion model can be adjusted according to the difference between the timbre characteristics and the text characteristics. In some embodiments, the module parameters of other modules in the singing voice conversion model except the audio encoding module and the text encoding module can be fixed, and the module parameters of the audio encoding module and the text encoding module can be adjusted according to the difference between the timbre characteristics and the text characteristics. When the training end condition is met, a trained singing voice conversion model can be obtained.

[0063] When the training of the singing voice conversion model is completed, since the timbre features and text features output by the audio encoding module and the text encoding module for the paired sample songs and singer's timbre description texts are already the same or relatively similar, the timbre description text can be input into the trained singing voice conversion model. The model extracts the text features corresponding to the input timbre description text through the text encoding module, and obtains the target timbre features based on the text features corresponding to the timbre text to perform song timbre conversion.

[0064] In the above-mentioned singing voice conversion model training method, the audio encoding module and the text encoding module in the singing voice conversion model are trained by using paired sample songs and singers' timbre descriptions, so that the text features output by the text encoding module can be made as close as possible to the timbre features output by the audio encoding module, so that after the training is completed, the text features can be used to replace the timbre features. When timbre conversion is required, it is only necessary to input any timbre description text related to the target timbre, without the need for users to record in advance or find sample recordings, thereby effectively improving the singing voice conversion efficiency.

[0065] In some exemplary embodiments, in step S103, adjusting the model parameters of the singing voice conversion model according to the difference between the timbre feature and the text feature to obtain a trained singing voice conversion model may include the following steps:

[0066] S1031, obtaining phoneme pronunciation feature distribution information of the sample song according to the text features and the pronunciation information of the sample song.

[0067] In a specific implementation, the pronunciation information corresponding to the sample song can be obtained. The pronunciation information is information that is independent of the singer and is related to the content of the sample song. In some examples, the pronunciation information of the sample song may include information related to the pitch of the sample song and the phonemes corresponding to the lyrics text. Among them, phoneme is the smallest unit of Chinese pronunciation. For example, the pronunciation units corresponding to the word "good" include "h" and "ao"; pitch is also called fundamental frequency. When the sound-emitting body makes a sound due to vibration, the sound can generally be decomposed into many simple sine waves. That is to say, all natural sounds can be understood as composed of many sine waves with different frequencies, among which the sine wave with the lowest frequency is the fundamental tone, and other sine waves with higher frequencies are overtones.

[0068] Since different singers may have different phoneme pronunciation methods when singing songs, in this step, after obtaining the text features of the singer's timbre description text, the text features and the pronunciation information of the sample song can be fused, and the phoneme pronunciation feature distribution information of the sample song can be obtained according to the fusion result, wherein the phoneme pronunciation feature distribution information can reflect the distribution of various phoneme pronunciation features. In some embodiments, the phoneme pronunciation feature distribution information can be a priori distribution of phoneme pronunciation features.

[0069] S1032, obtaining audio feature distribution information used to decode the sample song, and the distribution mapping module in the singing voice conversion model performs distribution mapping on the phoneme pronunciation feature distribution information and the audio feature distribution information to obtain a distribution mapping result.

[0070] In practical applications, the embodiments of the present application can also obtain audio feature distribution information for decoding sample songs, wherein the audio feature distribution information can characterize the distribution of audio features, which can be understood as high-dimensional features obtained from the latent space, and the audio feature distribution information can also be called the latent variable of the sample song. In some embodiments, the audio feature distribution information includes the posterior distribution of the audio feature. By decoding the encoded audio feature distribution information, the waveform corresponding to the singing signal can be obtained.

[0071] In one embodiment, obtaining audio feature distribution information for decoding a sample song may include the following steps: obtaining a spectrogram corresponding to the sample song; inputting the spectrogram into a posterior distribution encoder to obtain a posterior distribution of audio features corresponding to the sample song output by the posterior distribution encoder; and using the posterior distribution of audio features as the audio feature distribution information of the sample song.

[0072] In a specific implementation, the spectrogram corresponding to the sample song can be obtained, such as a linear-scale spectrogram. The spectrogram can then be input into the posterior distribution encoder in the singing voice conversion model to obtain the posterior distribution of the audio features corresponding to the sample song output by the posterior distribution encoder as the audio feature distribution information of the sample song.

[0073] Although the distribution information of phoneme pronunciation features and audio feature distribution information are both distribution information, the phoneme pronunciation features are mainly related to the timbre of a specific singer and the pronunciation information of phonemes, pitches, etc. involved in the sample songs. Phoneme pronunciation features are relatively stable and simple, while audio features cover information of multiple dimensions of audio signals, such as spectrum characteristics, time domain characteristics, cepstrum characteristics, etc. These information are numerous in quantity and have complex correlations with each other. Since the audio signal itself is highly complex and diverse, the distribution of audio features also presents greater complexity. It can be understood that there is a certain difference in the complexity of the distribution information of phoneme pronunciation features and audio feature distribution information, and the audio feature distribution information is more complex than the phoneme pronunciation feature distribution information.

[0074] Since the audio feature distribution information can decode the sample song, after the phoneme pronunciation feature distribution information of the sample song is generated according to the text features and the pronunciation information of the sample song, the singing voice conversion model can try to construct a distribution mapping relationship between the phoneme pronunciation feature distribution information and the audio feature distribution information through the distribution mapping module in the model, so that the distribution mapping relationship can be used to perform distribution mapping between the phoneme pronunciation feature distribution information and the audio feature distribution information to obtain a distribution mapping result. In some exemplary embodiments, the distribution mapping module can be a processing module using a normalizing flow technology, such as a flow model, and the distribution mapping module converts a simple prior distribution into a more complex posterior distribution by reversibly transforming it, thereby improving the expressive power of the prior distribution and better representing the distribution characteristics of the real sample.

[0075] It can be understood that when the module parameters of the distribution mapping module are appropriate, the distribution mapping module can correctly map simple phoneme pronunciation feature distribution information into more complex audio feature distribution information, thereby decoding the corresponding song using the mapped audio feature distribution information.

[0076] S1033, adjusting the model parameters of the singing voice conversion model according to the difference between the timbre features and the text features, and the difference between the distribution mapping result and the reference distribution mapping result, to obtain a trained singing voice conversion model.

[0077] Specifically, the embodiment of the present application can also obtain a reference distribution mapping result, which can be a predetermined, correct distribution mapping. By comparing the distribution mapping result output by the distribution mapping module in the singing voice conversion model with the reference distribution mapping result, it can be determined whether the current module parameters of the distribution mapping module are appropriate, thereby adjusting the singing voice conversion model accordingly.

[0078] Then, according to the difference between the timbre features and the text features, as well as the difference between the distribution mapping results and the reference distribution mapping results, the model parameters of the singing voice conversion model are adjusted. For example, according to the difference between the timbre features and the text features, as well as the difference between the distribution mapping results and the reference distribution mapping results, the model loss value is determined, and the model parameters are iteratively adjusted according to the model loss value. When the training end conditions are met, the trained singing voice conversion model is obtained.

[0079] In some optional embodiments, each module in the singing voice conversion model can be trained at the same time, for example, the text encoding module, audio encoding module and distribution mapping module in the singing voice conversion model can be trained at the same time, or different modules in the singing voice conversion model can be trained one after another, for example, the module parameters of other modules in the model except the timbre encoding module and the text encoding module can be fixed first, and then the module parameters of the text encoding module and the audio encoding module can be adjusted according to the difference between the timbre characteristics and the text characteristics. When the training end condition is met, the timbre encoding module and the text encoding module are fixed again, and the other modules are trained. It can be understood that whether the modules are trained at the same time or different modules are trained one after another, the model parameters of the singing voice conversion model can be adjusted according to the difference between the timbre characteristics and the text characteristics, and the difference between the distribution mapping results and the reference distribution mapping results.

[0080] In this embodiment, based on the existing text features and sample song pronunciation information, a distribution mapping relationship between the simple phoneme pronunciation feature distribution mapping corresponding to the singer and the audio feature distribution can be constructed, which helps the singing voice conversion model to generate more natural and expressive singing audio in combination with the timbre description text input by the user, and enhance the diversity and naturalness of the singing voice conversion process.

[0081] In one embodiment, in step S1033, adjusting the model parameters of the singing voice conversion model according to the difference between the timbre feature and the text feature, and the difference between the distribution mapping result and the reference distribution mapping result, may include the following steps:

[0082] The distribution decoding module in the singing voice conversion model decodes the corresponding predicted song according to the audio feature distribution information; determines the first loss value according to the difference between the timbre feature and the text feature, determines the second loss value according to the difference between the distribution mapping result and the reference distribution mapping result, and determines the third loss value according to the difference between the predicted song and the sample song; determines the model loss value according to the first loss value, the second loss value and the third loss value, and adjusts the model parameters of the singing voice conversion model according to the model loss value.

[0083] In one embodiment, Figure 2 As shown, the structure of a singing voice conversion model is shown, which includes a text encoder and an audio encoder for encoding the timbre features of the singer, a stream model as a distribution mapping module, and a distribution encoder and a distribution decoder. For the paired wav format singing dry voice data (i.e., sample songs) and the singer's timbre description text, the corresponding timbre features and text features can be obtained through the audio encoder and the text encoder respectively.

[0084] Since the audio waveform is decoded by the audio feature distribution information in this embodiment, the correct decoding of the audio feature distribution information will affect the conversion effect of the song timbre. In this regard, the distribution decoding module in the singing voice conversion model can decode the corresponding predicted song according to the audio feature distribution information, and then determine the first loss value according to the difference between the timbre feature and the text feature, determine the second loss value according to the difference between the distribution mapping result and the reference distribution mapping result, and determine the third loss value according to the difference between the predicted song and the sample song.

[0085] Among them, the first loss value is positively correlated with the difference between the timbre feature and the text feature, the second loss value is positively correlated with the difference between the distribution mapping result and the reference distribution mapping result, and the difference between the predicted song and the sample song is positively correlated with the third loss value. In some exemplary embodiments, the first loss value of the timbre feature and the text feature can be determined according to a regression loss function (such as a mean squared error loss function (MSELoss)), and the difference between the predicted song and the sample song can be determined according to the difference between the audio waveform of the predicted song and the audio waveform of the sample song.

[0086] Furthermore, the model loss value can be determined according to the first loss value, the second loss value and the third loss value, and the model parameters of the singing voice conversion model can be adjusted according to the model loss value, for example, based on the back propagation algorithm, the model parameters can be adjusted according to the model loss value.

[0087] In this embodiment, by determining the model loss value based on the first loss value, the second loss value and the third loss value, on the one hand, the features output by the text encoding module and the audio encoding module can be made as close as possible, so that the text features output by the text encoding module can subsequently replace the timbre features output by the audio encoding module; on the other hand, the distribution mapping relationship between the phoneme feature distribution information and the audio feature distribution information can be effectively constructed, so that a complex audio feature distribution can be fitted according to a simple phoneme feature distribution, thereby improving the naturalness and details of the synthesized singing voice; on the other hand, the decoding accuracy of the distribution decoder can be effectively improved, making the decoding result more accurate.

[0088] In one embodiment, the distribution decoding module in the singing voice conversion model decodes the corresponding predicted song according to the audio feature distribution information, which may include the following steps:

[0089] The audio feature distribution information, text features and pitch information of the sample song are input into the distribution decoding module in the singing voice conversion model, and the predicted song is obtained according to the output result of the distribution decoding module.

[0090] In practical applications, when singers with different timbres sing the same song, there will be some differences in the audio waveform of the song. In this regard, in this embodiment, in addition to inputting the audio feature distribution information into the distribution decoding module, the text features and the pitch information of the sample song can also be input into the distribution decoding module together, and the distribution decoding module decodes the predicted song by combining the above three types of information. By inputting the text features and the pitch information of the sample song into the distribution decoding module together, the feature information related to the singer's timbre and the song pitch can be enhanced during the decoding process, which helps to improve the correspondence between the predicted song obtained by decoding and the singer's timbre and the song pitch.

[0091] In one embodiment, the reference distribution mapping result includes phoneme pronunciation feature distribution information or audio feature distribution information; in step S1033, the difference between the distribution mapping result and the reference distribution mapping result can be determined by the following steps:

[0092] If the distribution mapping result is the audio feature distribution mapping result obtained by the distribution mapping module through forward mapping the audio feature distribution information, the difference between the audio feature distribution mapping result and the phoneme pronunciation feature distribution information is determined; if the distribution mapping result is the phoneme feature distribution mapping result obtained by the distribution mapping module through reverse mapping the phoneme pronunciation feature distribution information, the difference between the phoneme feature distribution mapping result and the audio feature distribution information is determined.

[0093] Among them, the audio feature distribution mapping result is obtained by the distribution mapping module performing forward mapping on the audio feature distribution information; the phoneme feature distribution mapping result is obtained by the distribution mapping module performing reverse mapping on the phoneme pronunciation feature distribution information.

[0094] In practical applications, the distribution mapping module in the singing voice conversion model is a reversible mapping module, that is, the distribution mapping module can map the phoneme pronunciation feature distribution information to the audio feature distribution information, and can also map the audio feature distribution information to the phoneme pronunciation feature distribution information.

[0095] After obtaining the phoneme pronunciation feature distribution information of the sample song according to the text features and pronunciation information, distribution mapping can be performed as needed. Specifically, if the distribution mapping module performs forward mapping on the audio distribution feature information, that is, maps the audio feature distribution information Za to Zt', then the mapped phoneme pronunciation feature distribution information can be used as the distribution mapping result, the phoneme pronunciation feature distribution information Zt generated according to the text features and pronunciation information can be used as the reference distribution mapping result, and the difference between the distribution mapping result Zt' and the phoneme pronunciation feature distribution information Zt can be determined.

[0096] If the distribution mapping module performs reverse mapping on the phoneme pronunciation feature distribution information, that is, maps the phoneme pronunciation feature distribution information Zt to the audio feature distribution information Za', then the mapped audio feature distribution information Za' can be used as the distribution mapping result, and the pre-acquired audio feature distribution information Za can be used as the reference distribution mapping result, and then the difference between the phoneme feature distribution mapping result Za' and the audio feature distribution information Za is determined.

[0097] In this embodiment, the distribution mapping module performs reversible distribution mapping on the phoneme feature distribution information and the audio feature distribution information, which enables the distribution mapping module to accurately construct the relationship between simple phoneme feature distribution information and complex audio feature distribution information, ensuring that key information will not be lost during the data transformation process, and effectively improving the performance and stability of the singing conversion model.

[0098] In some exemplary embodiments, the pronunciation information of the sample song includes pitch information of the sample song and phoneme features of the sample song; in step S1031, obtaining the phoneme pronunciation feature distribution information of the sample song according to the text features and the pronunciation information of the sample song may include the following steps:

[0099] The text features, pitch information of the sample songs and phoneme features of the sample songs are input into the text encoding module to obtain the phoneme prior distribution output by the text encoding module; the phoneme prior distribution is used as the phoneme pronunciation feature distribution information of the sample songs.

[0100] In practical applications, the pitch information of the sample song and the phoneme features of the sample song can be obtained. In some exemplary embodiments, the audio signal of the sample song can be converted into the time-frequency domain using a frequency domain analysis method such as Short-Time Fourier Transform (STFT), thereby extracting the pitch information, or Linear Predictive Cepstral Coefficient (LPCC) or Mel Frequency Cepstrum Coefficient (MFCC) can be used to extract the pitch information; the phoneme prior probability of the sample song can also be obtained through a phoneme feature extraction model as a phoneme feature.

[0101] Then, the text features, the pitch information and the phoneme features of the sample song can be input into the text encoding module, and the text encoding module obtains the phoneme prior distribution output by the text encoding module according to the text features, the pitch information and the phoneme features, and uses the phoneme prior distribution as the phoneme pronunciation feature distribution information of the sample song. In this embodiment, by inputting the pitch information and the phoneme features of the sample song into the text encoding module together, the acoustic features of the singer when making the sound of the corresponding phoneme according to the pitch of the sample song can be effectively simulated, providing a basis for the subsequent timbre conversion.

[0102] In one embodiment, Figure 3 As shown, a song timbre conversion method is provided. This embodiment takes the method applied to a terminal as an example for illustration. It can be understood that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0103] S301, obtaining a song to be converted and timbre configuration information including at least a timbre description text.

[0104] In a specific implementation, the user can obtain songs to be converted to timbre, i.e. songs to be converted. In some examples, the songs sung by the user may be out of tune or lack singing skills. In this case, songs sung by other singers with singing skills or accurate pitch can be used as songs to be converted.

[0105] On the other hand, the timbre configuration information can also be set, wherein the timbre configuration information at least includes a timbre description text. The timbre description text can be a natural language sentence arbitrarily input according to the user's timbre setting requirements. The natural language sentence can describe the characteristics of the singer's timbre, such as "bright", "magnetic", "round", "delicate" and other descriptive texts.

[0106] S302, input the pronunciation information and timbre configuration information of the song to be converted into a trained singing voice conversion model, and obtain the song after the singing voice conversion model outputs the timbre conversion.

[0107] The singing voice conversion model is trained according to the singing voice conversion model training method in any of the above embodiments.

[0108] In this step, the timbre configuration information and the pronunciation information corresponding to the song to be converted can be input into the trained singing voice conversion model together, and the singing voice conversion model outputs the song after the timbre conversion. In some embodiments, the pronunciation information corresponding to the song to be converted can be extracted by the singing voice conversion model, or the pronunciation information can be extracted by other models.

[0109] For example, Figure 4a As shown, the pitch information, phoneme feature PPG (Phonetic Posteriorgrams) and timbre configuration information of the song to be converted can be input into the singing voice conversion model. For the timbre description text in the timbre configuration information, the text encoding module can obtain the corresponding text features, and then the text features corresponding to the timbre description text are used as target timbre features and input into the text encoding module together with the pronunciation information corresponding to the song to be converted, to obtain the phoneme pronunciation feature distribution information output by the text encoding module, and then the distribution mapping module is used to convert the phoneme pronunciation feature distribution information into audio feature distribution information, and after being decoded by the distribution decoder, the song after timbre conversion can be obtained.

[0110] In this embodiment, the song to be converted and the timbre configuration information including at least the timbre description text can be obtained; then the pronunciation information and timbre configuration information of the song to be converted are input into the trained singing voice conversion model, and the singing voice conversion model outputs the song after the timbre conversion. In this embodiment, by inputting the timbre description text, the trained singing voice conversion model can determine the target timbre features using the text features corresponding to the timbre description text. When the timbre conversion is required, it is only necessary to input any timbre description text related to the target timbre, without the need for the user to record in advance or find sample recordings, which effectively improves the singing voice conversion efficiency.

[0111] Through the song timbre conversion method of the present application, the user's out-of-tune singing or singing lacking singing skills can be transformed into singing skills; in addition, the present application can efficiently use natural language input to fine-tune the synthesized timbre, thereby achieving the effect of beautifying the voice.

[0112] In one embodiment, the timbre configuration information also includes material audio with a specified timbre; in step S302, the pronunciation information and timbre configuration information of the song to be converted are input into a trained singing voice conversion model, and the singing voice conversion model outputs a song after the timbre is converted, which may include the following steps:

[0113] S3021, the text encoding module in the singing voice conversion model performs feature extraction on the timbre description text to obtain text features of the timbre description text, and the audio encoding module in the singing voice conversion model performs feature extraction on the material audio to obtain timbre features of the material audio.

[0114] like Figure 4b As shown, in some embodiments, the phoneme configuration information may include both the timbre description text and the material audio with the specified timbre. After the phoneme configuration information is input into the singing voice conversion model, on the one hand, the text encoding module in the singing voice conversion model may perform feature extraction on the timbre description text to obtain the text features of the timbre description text, and on the other hand, the audio encoding module in the singing voice conversion model may perform feature extraction on the material audio to obtain the timbre features of the material audio.

[0115] S3022: Based on the feature weights of the text features of the timbre description text and the timbre features of the material audio, feature fusion is performed on the text features of the timbre description text and the timbre features of the material audio to obtain target timbre features.

[0116] After obtaining the text features of the timbre description text and the timbre features of the material audio, their respective feature weights can be obtained for fusion, and the fused features can be used as the target timbre features. For example, if the text features of the timbre description text and the timbre features of the material audio are fused with a feature weight of 3:7, the target timbre features = 0.3*text features of the timbre description text+0.7*timbre features of the material audio.

[0117] S3023, outputting the song after timbre conversion according to the target timbre characteristics and the pronunciation information of the song to be converted.

[0118] After obtaining the target timbre features, the singing voice conversion model can output the song after timbre conversion according to the target timbre features and the pronunciation information of the song to be converted. The specific output process can refer to the above embodiment and will not be described in detail here.

[0119] In this embodiment, the text features of the timbre description text and the timbre features of the material audio are fused according to their respective feature weights to obtain the target timbre features. The singer's timbre can be adjusted through both text and audio at the same time, greatly improving the flexibility and efficiency of singing voice timbre conversion.

[0120] In some other embodiments, the timbre configuration information may also include only the material audio, for example, Figure 4cAs shown, the target timbre feature can be determined only according to the timbre feature output by the audio encoding module. Those skilled in the art can select the type of timbre configuration information input into the singing voice conversion model according to actual conditions.

[0121] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0122] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 5 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store singing dry voice data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a singing conversion model training method or a song timbre conversion method is implemented.

[0123] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Figure 6As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be realized through WIFI, a mobile cellular network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, a singing voice conversion model training method or a song timbre conversion method is realized. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0124] Those skilled in the art will understand that Figure 5 and Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0125] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0126] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0127] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0128] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0129] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0130] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0131] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A singing voice conversion model training method, characterized in that: The method comprises: Obtaining a sample song and a voice description text of a singer paired with the sample song; The audio encoding module in the singing voice conversion model obtains the timbre characteristics of the sample song, and the text encoding module in the singing voice conversion model obtains the text characteristics corresponding to the timbre description text of the singer; According to the difference between the timbre features and the text features, the model parameters of the singing voice conversion model are adjusted to obtain a trained singing voice conversion model; the trained singing voice conversion model is used to extract the text features corresponding to the input timbre description text through the text encoding module, and obtain the target timbre features according to the text features corresponding to the timbre description text to perform song timbre conversion.

2. The method according to claim 1, characterized in that The method of adjusting the model parameters of the singing voice conversion model according to the difference between the timbre feature and the text feature to obtain a trained singing voice conversion model comprises: Acquire phoneme pronunciation feature distribution information of the sample song according to the text feature and the pronunciation information of the sample song; Acquire audio feature distribution information used to decode the sample song, and perform distribution mapping on the phoneme pronunciation feature distribution information and the audio feature distribution information by a distribution mapping module in the singing voice conversion model to obtain a distribution mapping result; According to the difference between the timbre feature and the text feature, and the difference between the distribution mapping result and the reference distribution mapping result, the model parameters of the singing voice conversion model are adjusted to obtain a trained singing voice conversion model.

3. The method according to claim 2, characterized in that The adjusting of the model parameters of the singing voice conversion model according to the difference between the timbre feature and the text feature, and the difference between the distribution mapping result and the reference distribution mapping result, comprises: The distribution decoding module in the singing voice conversion model decodes the corresponding predicted song according to the audio feature distribution information; Determine a first loss value according to the difference between the timbre feature and the text feature, determine a second loss value according to the difference between the distribution mapping result and the reference distribution mapping result, and determine a third loss value according to the difference between the predicted song and the sample song; A model loss value is determined according to the first loss value, the second loss value and the third loss value, and a model parameter of the singing voice conversion model is adjusted according to the model loss value.

4. The method according to claim 3, characterized in that The distribution decoding module in the singing voice conversion model decodes the corresponding predicted song according to the audio feature distribution information, including: The audio feature distribution information, the text features and the pitch information of the sample song are input into the distribution decoding module in the singing voice conversion model, and the predicted song is obtained according to the output result of the distribution decoding module.

5. The method according to claim 2, characterized in that: The reference distribution mapping result includes the phoneme pronunciation feature distribution information or the audio feature distribution information; The difference between the distribution mapping result and the reference distribution mapping result is determined by the following steps: If the distribution mapping result is an audio feature distribution mapping result obtained by the distribution mapping module performing forward mapping on the audio feature distribution information, determining a difference between the audio feature distribution mapping result and the phoneme pronunciation feature distribution information; If the distribution mapping result is a phoneme feature distribution mapping result obtained by the distribution mapping module performing reverse mapping on the phoneme pronunciation feature distribution information, a difference between the phoneme feature distribution mapping result and the audio feature distribution information is determined.

6. The method according to any one of claims 1 to 5, characterized in that The pronunciation information of the sample song includes pitch information of the sample song and phoneme features of the sample song; The step of obtaining the phoneme pronunciation feature distribution information of the sample song according to the text feature and the pronunciation information of the sample song includes: Inputting the text features, the pitch information of the sample song and the phoneme features of the sample song into the text encoding module to obtain a phoneme prior distribution output by the text encoding module; The phoneme prior distribution is used as the phoneme pronunciation feature distribution information of the sample song.

7. A song timbre conversion method, characterized in that: The method comprises: Obtaining a song to be converted and timbre configuration information including at least timbre description text; Input the pronunciation information of the song to be converted and the timbre configuration information into a trained singing voice conversion model, and obtain the song after the timbre conversion is output by the singing voice conversion model; Wherein, the singing voice conversion model is trained according to the singing voice conversion model training method as described in any one of claims 1 to 6.

8. The method according to claim 7, characterized in that The timbre configuration information also includes material audio with a specified timbre; The step of inputting the pronunciation information of the song to be converted and the timbre configuration information into a trained singing voice conversion model to obtain the song after the singing voice conversion model outputs the timbre conversion, comprising: The text encoding module in the singing voice conversion model performs feature extraction on the timbre description text to obtain text features of the timbre description text, and the audio encoding module in the singing voice conversion model performs feature extraction on the material audio to obtain timbre features of the material audio; According to the feature weights of the text features of the timbre description text and the timbre features of the material audio, feature fusion is performed on the text features of the timbre description text and the timbre features of the material audio to obtain a target timbre feature; The song after timbre conversion is output according to the target timbre feature and the pronunciation information of the song to be converted.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the singing voice conversion model training method described in any one of claims 1 to 6 or the steps of the song timbre conversion method described in any one of claims 7 to 8 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the singing voice conversion model training method described in any one of claims 1 to 6 or the steps of the song timbre conversion method described in any one of claims 7 to 8 are implemented.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the singing voice conversion model training method described in any one of claims 1 to 6 or the steps of the song timbre conversion method described in any one of claims 7 to 8 are implemented.

Images