A training method, apparatus, medium, and electronic equipment for a singing voice synthesis model.

Abstract

This specification discloses a training method, apparatus, medium, and electronic device for a vocal synthesis model, comprising: training a vocal synthesis model to be trained using pre-collected song data. During training, the phonemes of the song are first determined based on the lyrics data. Then, the phonemes are concatenated with the musical score data, and the concatenation result is input into the melody prediction layer of the vocal synthesis model to be trained, obtaining the first melody distribution of the song. Next, the first melody score of the determined song is input into the vocal prediction layer to obtain the predicted audio and a second melody distribution. Then, based on the difference between the first and second melody distributions, and the difference between the audio data and the predicted audio, a target loss is determined, and the vocal synthesis model to be trained is trained with the minimum target loss as the optimization objective. Increasing the sampling rate of the trained vocal synthesis model reduces vibrato in the vocals generated by the model.

Description

Technical Field

[0001] This specification relates to the field of computer technology, and in particular to a training method, apparatus, medium, and electronic device for a singing voice synthesis model. Background Technology

[0002] With the continuous development of technology, human-computer interaction is becoming more and more frequent and intelligent. Among them, the synthesis of singing voice through human-computer interaction has received widespread attention.

[0003] Generally, vocal synthesis refers to the process of synthesizing vocals based on lyrics and sheet music. Therefore, how to synthesize vocals is a very important question.

[0004] Based on this, this specification provides a training method for a singing voice synthesis model. Summary of the Invention

[0005] This specification provides a training method, apparatus, medium, and electronic device for a singing voice synthesis model, in order to partially solve the aforementioned problems existing in the prior art.

[0006] The following technical solution is adopted in this specification:

[0007] This specification provides a training method for a singing voice synthesis model, wherein the singing voice synthesis model to be trained includes a melody prediction layer and a singing voice prediction layer; the method includes:

[0008] Acquire song data from pre-collected songs, wherein the song data includes at least lyrics data, sheet music data, and audio data;

[0009] Based on the lyrics data contained in the song data, determine the phonemes corresponding to the lyrics data;

[0010] The musical score data and the phonemes are concatenated to obtain the first result;

[0011] The first result is input into the melody prediction layer of the singing synthesis model to be trained to predict the first melody distribution of the song, wherein the first melody distribution includes information about the phonemes corresponding to the lyrics data.

[0012] Based on the audio data, the first Mel spectrum of the song is determined, and the first Mel spectrum is input into the singing prediction layer of the singing synthesis model to be trained to obtain the predicted audio and the second melody distribution.

[0013] Based on the difference between the first melody distribution and the second melody distribution, and the difference between the audio data and the predicted audio, a target loss is determined, and the singing synthesis model to be trained is trained with the minimum target loss as the optimization objective.

[0014] Optionally, the melody prediction layer includes a feature extraction layer, a duration prediction layer, a fundamental frequency prediction layer, a Mel spectrum prediction layer, and a distribution prediction layer;

[0015] The first result is input into the melody prediction layer of the vocal synthesis model to be trained to obtain the first melody distribution of the song, specifically including:

[0016] The first result is input into the feature extraction layer of the melody prediction layer to obtain the implicit features of the phoneme;

[0017] The implicit features are input into the duration prediction layer of the melody prediction layer to obtain the first pronunciation duration of the phoneme.

[0018] Based on the first pronunciation duration of the phoneme, the implicit features of the phoneme are expanded to obtain a second result;

[0019] The second result is input into the fundamental frequency prediction layer of the melody prediction layer to obtain the first fundamental frequency of the audio frame of the song;

[0020] The second result is concatenated with the first fundamental frequency of the audio frame of the song to obtain the third result;

[0021] The third result is input into the Mel score prediction layer of the melody prediction layer to obtain the second Mel score of the song;

[0022] The second Mel spectrum and the second result are concatenated to obtain the fourth result;

[0023] The fourth result is input into the distribution prediction layer of the melody prediction layer to obtain the first melody distribution of the song.

[0024] Optionally, the singing prediction layer includes a spectrogram distribution prediction layer, a conversion layer, and a differentiable digital signal processing layer;

[0025] Based on the audio data, a first Mel score of the song is determined, and the first Mel score is input into the singing prediction layer of the singing synthesis model to be trained to obtain the predicted audio and a second melody distribution, specifically including:

[0026] Based on the audio data, a preset first algorithm is used to determine the first Mel score of the song;

[0027] The first Mel spectrum is input into the spectral distribution prediction layer of the singing prediction layer to obtain the predicted spectral distribution of the song;

[0028] The predicted spectral distribution of the song is input into the differentiable digital signal processing layer of the vocal prediction layer to obtain the predicted audio of the song, and the predicted spectral distribution is input into the transformation layer of the vocal prediction layer to obtain the second melody distribution.

[0029] Optionally, based on the difference between the first melody distribution and the second melody distribution, and the difference between the audio data and the predicted audio, a target loss is determined, specifically including:

[0030] A first loss is determined based on the difference between the audio data and the predicted audio, and a second loss is determined based on the difference between the first melody distribution and the second melody distribution;

[0031] The sum of the first loss and the second loss is taken as the target loss.

[0032] Optionally, based on the difference between the first melody distribution and the second melody distribution, and the difference between the audio data and the predicted audio, a target loss is determined, specifically including:

[0033] Based on the audio data, a preset second algorithm is used to determine the second pronunciation duration of the phoneme;

[0034] The third loss is determined based on the difference between the first pronunciation duration and the second pronunciation duration;

[0035] The target loss is determined based on the difference between the first melody distribution and the second melody distribution, the difference between the audio data and the predicted audio, and the third loss.

[0036] Optionally, based on the difference between the first melody distribution and the second melody distribution, and the difference between the audio data and the predicted audio, a target loss is determined, specifically including:

[0037] Based on the audio data, a preset third algorithm is used to determine the second fundamental frequency of the audio frames of the song;

[0038] The fourth loss is determined based on the difference between the first fundamental frequency and the second fundamental frequency;

[0039] The target loss is determined based on the difference between the first melody distribution and the second melody distribution, the difference between the audio data and the predicted audio, and the fourth loss.

[0040] Optionally, based on the difference between the first melody distribution and the second melody distribution, and the difference between the audio data and the predicted audio, a target loss is determined, specifically including:

[0041] The fifth loss is determined based on the difference between the second Mel spectrum and the first Mel spectrum;

[0042] The target loss is determined based on the difference between the first melody distribution and the second melody distribution, the difference between the audio data and the predicted audio, and the fifth loss.

[0043] Optionally, based on the difference between the first melody distribution and the second melody distribution, and the difference between the audio data and the predicted audio, a target loss is determined, specifically including:

[0044] Based on the audio data, a preset algorithm is used to determine the second pronunciation duration of the phoneme and the second fundamental frequency of the audio frame of the song;

[0045] A third loss is determined based on the difference between the first and second pronunciation durations; a fourth loss is determined based on the difference between the first and second fundamental frequencies; and a fifth loss is determined based on the difference between the second and first Mel spectra.

[0046] The target loss is determined based on the difference between the first melody distribution and the second melody distribution, the difference between the audio data and the predicted audio, the third loss, the fourth loss, and the fifth loss.

[0047] Optionally, the predicted spectral distribution of the song is input into the differentiable digital signal processing layer of the vocal prediction layer to obtain the predicted audio of the song, specifically including:

[0048] The predicted spectral distribution of the song is input into the differentiable digital signal processing layer of the vocal prediction layer to determine the harmonic parameters and noise parameters of the audio frames of the song.

[0049] For each audio frame of the song, determine the predicted fundamental frequency corresponding to that audio frame;

[0050] Based on the predicted fundamental frequency corresponding to the audio frame and the harmonic parameters of the audio frame, the harmonic signal of the audio frame is determined, and based on the predicted fundamental frequency corresponding to the audio frame and the noise parameters of the audio frame, the noise signal of the audio frame is determined.

[0051] The sum of the harmonic signal and the noise signal of the audio frame is determined as the audio signal of the audio frame;

[0052] The predicted audio of the song is determined based on the audio signals of each audio frame.

[0053] Optionally, the method further includes:

[0054] In response to user input, determine the lyrics data and sheet music data of the song to be synthesized input by the user;

[0055] The phonemes corresponding to the lyrics data input by the user are determined as the phonemes to be synthesized;

[0056] The phonemes to be synthesized are concatenated with the sheet music data input by the user to obtain the first result;

[0057] The first result is input into the melody prediction layer of the trained singing synthesis model to predict the melody distribution of the song to be synthesized.

[0058] The melody distribution of the song to be synthesized is input into the transformation layer of the singing prediction layer of the trained singing synthesis model to obtain the spectral distribution of the song to be synthesized.

[0059] The spectral distribution of the song to be synthesized is input into the differentiable digital signal processing layer in the singing prediction layer of the trained singing synthesis model to obtain the audio of the song to be synthesized.

[0060] Optionally, the melody prediction layer includes a feature extraction layer, a duration prediction layer, a fundamental frequency prediction layer, a Mel spectrum prediction layer, and a distribution prediction layer;

[0061] The method further includes:

[0062] In response to user input, determine the lyrics data and sheet music data of the song to be synthesized input by the user;

[0063] The phonemes corresponding to the lyrics data input by the user are determined as the phonemes to be synthesized;

[0064] The phonemes to be synthesized are combined with the sheet music data input by the user.

[0065] The spliced ​​result is input into the feature extraction layer of the melody prediction layer of the trained singing synthesis model to obtain the implicit features of the phonemes to be synthesized.

[0066] The implicit features of the phoneme to be synthesized are input into the duration prediction layer to obtain the pronunciation duration of the phoneme to be synthesized.

[0067] Based on the pronunciation duration of the phoneme to be synthesized, the implicit features of the phoneme to be synthesized are expanded to obtain the expanded result;

[0068] The augmentation result is input into the fundamental frequency prediction layer to obtain the fundamental frequency of the audio frame of the song to be synthesized;

[0069] The augmented result and the fundamental frequency are spliced ​​together to obtain the first splicing result;

[0070] The first splicing result is input into the Mel spectrum prediction layer to obtain the Mel spectrum of the song to be synthesized;

[0071] The Mel spectrum and the augmented result are spliced ​​together to obtain a second splicing result;

[0072] The second splicing result is input into the distribution prediction layer to obtain the melody distribution of the song to be synthesized;

[0073] The melody distribution is input into the transformation layer of the vocal prediction layer of the trained vocal synthesis model to obtain the spectrogram distribution of the song to be synthesized.

[0074] The spectral distribution is input into the differentiable digital signal processing layer to obtain the audio of the song to be synthesized.

[0075] This specification provides a training device for a singing voice synthesis model. The singing voice synthesis model to be trained includes a melody prediction layer and a singing voice prediction layer; the device includes:

[0076] The acquisition module is used to acquire song data of pre-collected songs, wherein the song data includes at least lyrics data, sheet music data and audio data;

[0077] The phoneme module is used to determine the phonemes corresponding to the lyrics data based on the lyrics data contained in the song data.

[0078] The determining module is used to concatenate the musical score data and the phonemes to obtain a first result;

[0079] The melody module is used to input the first result into the melody prediction layer of the singing synthesis model to be trained, and to predict the predicted melody distribution of the song, wherein the predicted melody distribution includes information on the phonemes corresponding to the lyrics data;

[0080] An audio module is used to determine the first melodic spectrum of the song based on the audio data, and input the first melodic spectrum into the singing prediction layer of the singing synthesis model to be trained to obtain the predicted audio and the second melody distribution.

[0081] The training module determines the target loss based on the difference between the first melody distribution and the second melody distribution, and the difference between the audio data and the predicted audio, and trains the singing synthesis model to be trained with the minimum target loss as the optimization objective.

[0082] Optionally, the melody prediction layer includes a feature extraction layer, a duration prediction layer, a fundamental frequency prediction layer, a Mel spectrum prediction layer, and a distribution prediction layer;

[0083] The melody module is specifically used to: input the first result into the feature extraction layer of the melody prediction layer to obtain the implicit features of the phoneme; input the implicit features into the duration prediction layer of the melody prediction layer to obtain the first pronunciation duration of the phoneme; expand the implicit features of the phoneme according to the predicted pronunciation duration of the phoneme to obtain a second result; input the second result into the fundamental frequency prediction layer of the melody prediction layer to obtain the first fundamental frequency of the audio frame of the song; concatenate the second result and the first fundamental frequency of the audio frame of the song to obtain a third result; input the third result into the Mel spectrum prediction layer of the melody prediction layer to obtain the second Mel spectrum of the song; concatenate the second Mel spectrum and the second result to obtain a fourth result; and input the fourth result into the distribution prediction layer of the melody prediction layer to obtain the first melody distribution of the song.

[0084] Optionally, the singing prediction layer includes a spectrogram distribution prediction layer, a conversion layer, and a differentiable digital signal processing layer;

[0085] The audio module is specifically used to: determine the first Mel spectrum of the song based on the audio data using a preset first algorithm; input the first Mel spectrum into the spectral distribution prediction layer of the vocal prediction layer to obtain the predicted spectral distribution of the song; input the predicted spectral distribution of the song into the differentiable digital signal processing layer of the vocal prediction layer to obtain the predicted audio of the song; and input the predicted spectral distribution into the conversion layer of the vocal prediction layer to obtain the second melody distribution.

[0086] Optionally, the training module is specifically configured to determine a first loss based on the difference between the audio data and the predicted audio, and to determine a second loss based on the difference between the first melody distribution and the second melody distribution; and to use the sum of the first loss and the second loss as the target loss.

[0087] Optionally, the training module is specifically used to: determine the second pronunciation duration of the phoneme based on the audio data using a preset second algorithm; determine a third loss based on the difference between the first pronunciation duration and the second pronunciation duration; and determine the target loss based on the difference between the first melody distribution and the second melody distribution, the difference between the audio data and the predicted audio, and the third loss.

[0088] Optionally, the training module is specifically used to: determine the second fundamental frequency of the audio frame of the song using a preset third algorithm based on the audio data; determine a fourth loss based on the difference between the first fundamental frequency and the second fundamental frequency; and determine the target loss based on the difference between the first melody distribution and the second melody distribution, the difference between the audio data and the predicted audio, and the fourth loss.

[0089] Optionally, the training module is specifically used to determine a fifth loss based on the difference between the second Mel spectrum and the first Mel spectrum; and to determine the target loss based on the difference between the first melody distribution and the second melody distribution, the difference between the audio data and the predicted audio, and the fifth loss.

[0090] Optionally, the training module is specifically used to: determine the second pronunciation duration of the phoneme and the second fundamental frequency of the audio frame of the song based on the audio data using a preset algorithm; determine a third loss based on the difference between the first pronunciation duration and the second pronunciation duration; determine a fourth loss based on the difference between the first fundamental frequency and the second fundamental frequency; determine a fifth loss based on the difference between the second Mel spectrum and the first Mel spectrum; and determine a target loss based on the difference between the first melody distribution and the second melody distribution, the difference between the audio data and the predicted audio, the third loss, the fourth loss, and the fifth loss.

[0091] Optionally, the audio module is specifically configured to: input the predicted spectral distribution of the song into the differentiable digital signal processing layer of the vocal prediction layer to determine the harmonic parameters and noise parameters of the audio frames of the song; for each audio frame of the song, determine the predicted fundamental frequency corresponding to that audio frame; determine the harmonic signal of the audio frame based on the predicted fundamental frequency and the harmonic parameters of that audio frame, and determine the noise signal of the audio frame based on the predicted fundamental frequency and the noise parameters of that audio frame; determine the sum of the harmonic signal and the noise signal of the audio frame as the audio signal of that audio frame; and determine the predicted audio of the song based on the audio signals of each audio frame.

[0092] Optionally, the device further includes:

[0093] An application module is used to respond to user input operations by determining the lyrics data and sheet music data of the song to be synthesized input by the user; determining the phonemes corresponding to the lyrics data input by the user as the phonemes to be synthesized; concatenating the phonemes to be synthesized with the sheet music data input by the user to obtain a first result; inputting the first result into the melody prediction layer of the trained vocal synthesis model to predict the melody distribution of the song to be synthesized; inputting the melody distribution of the song to be synthesized into the conversion layer in the vocal prediction layer of the trained vocal synthesis model to obtain the spectral distribution of the song to be synthesized; and inputting the spectral distribution of the song to be synthesized into the differentiable digital signal processing layer in the vocal prediction layer of the trained vocal synthesis model to obtain the audio of the song to be synthesized.

[0094] Optionally, the melody prediction layer includes a feature extraction layer, a duration prediction layer, a fundamental frequency prediction layer, a Mel spectrum prediction layer, and a distribution prediction layer;

[0095] The application module is used to respond to user input operations by determining the lyrics and sheet music data of the song to be synthesized; determining the phonemes corresponding to the lyrics data as the phonemes to be synthesized; concatenating the phonemes to be synthesized with the sheet music data; inputting the concatenated result into the feature extraction layer of the melody prediction layer of the trained vocal synthesis model to obtain the latent features of the phonemes to be synthesized; inputting the latent features of the phonemes to be synthesized into the duration prediction layer to obtain the pronunciation duration of the phonemes to be synthesized; expanding the latent features of the phonemes to be synthesized based on the pronunciation duration to obtain the expanded result; and inputting the expanded result into the feature extraction layer. The fundamental frequency of the audio frame of the song to be synthesized is obtained by inputting the fundamental frequency prediction layer; the augmentation result and the fundamental frequency are concatenated to obtain a first concatenation result; the first concatenation result is input into the Mel spectrum prediction layer to obtain the Mel spectrum of the song to be synthesized; the Mel spectrum and the augmentation result are concatenated to obtain a second concatenation result; the second concatenation result is input into the distribution prediction layer to obtain the melody distribution of the song to be synthesized; the melody distribution is input into the transformation layer of the vocal prediction layer of the trained vocal synthesis model to obtain the spectral distribution of the song to be synthesized; the spectral distribution is input into the differentiable digital signal processing layer to obtain the audio of the song to be synthesized.

[0096] This specification provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the training method for the above-described singing voice synthesis model.

[0097] This specification provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the training method of the above-described singing voice synthesis model.

[0098] The above-mentioned technical solutions adopted in this specification can achieve the following beneficial effects:

[0099] The training method for the vocal synthesis model provided in this manual involves acquiring pre-collected song data and determining the phonemes corresponding to the lyrics data. The score data and phonemes are then concatenated to obtain a first result. This first result is then input into the melody prediction layer of the vocal synthesis model to be trained, yielding the first melody distribution of the song. Next, based on the audio data, the first melodic spectrum of the song is determined and input into the vocal prediction layer of the vocal synthesis model to obtain the predicted audio and a second melody distribution. Finally, based on the differences between the first and second melody distributions, and the differences between the audio data and the predicted audio, a target loss is determined, and the vocal synthesis model is trained with the minimum target loss as the optimization objective.

[0100] As can be seen from the above method, this application trains the vocal synthesis model to be trained by using pre-collected song data when synthesizing vocals. During training, the phonemes of the song are first determined based on the lyrics data, and then the phonemes are concatenated with the score data. The concatenation result is input into the melody prediction layer of the vocal synthesis model to be trained, resulting in the first melody distribution of the song. Next, the first melodic spectrum of the song is determined based on the audio data, and the first melodic spectrum is input into the vocal prediction layer of the vocal synthesis model to be trained, resulting in the predicted audio and the second melody distribution. Then, based on the difference between the first and second melody distributions, and the difference between the audio data and the predicted audio, the target loss is determined. The vocal synthesis model to be trained is then trained based on the target loss, increasing the sampling rate of the trained vocal synthesis model, thereby reducing vibrato in the vocals generated by the model. Attached Figure Description

[0101] The accompanying drawings, which are included to provide a further understanding of this specification and form part of this specification, illustrate exemplary embodiments and are used to explain this specification, but do not constitute an undue limitation thereof. In the drawings:

[0102] Figure 1 This is a flowchart illustrating a training method for a singing voice synthesis model provided in this specification.

[0103] Figure 2 This is a schematic diagram of the structure of a melody prediction layer provided in this specification;

[0104] Figure 3 This is a schematic diagram of the structure of a singing prediction layer provided in this specification.

[0105] Figure 4 This is a schematic diagram of the structure of a singing voice synthesis model provided in this specification;

[0106] Figure 5A schematic diagram of the structure of a training device for a singing voice synthesis model provided in this specification;

[0107] Figure 6 This specification provides a corresponding Figure 1 A schematic diagram of the structure of an electronic device. Detailed Implementation

[0108] To make the objectives, technical solutions, and advantages of this specification clearer, the technical solutions of this specification will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of them. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this specification.

[0109] The technical solutions provided in the various embodiments of this specification are described in detail below with reference to the accompanying drawings.

[0110] Figure 1 This is a flowchart illustrating a training method for a singing voice synthesis model provided in this specification, including the following steps:

[0111] S100: Obtain song data from pre-collected songs, wherein the song data includes at least lyrics data, sheet music data, and audio data.

[0112] In this manual, the device used to train the vocal synthesis model acquires song data from pre-collected songs. This device can be a server or an electronic device such as a desktop computer or laptop. For ease of description, the training method for the vocal synthesis model provided in this manual will be explained below using a server as the primary execution device.

[0113] The pre-collected songs are those containing only vocals. The song data includes at least lyrics, sheet music, and audio data, and may also include singer information. The lyrics data is the text data of the lyrics corresponding to the song. The sheet music data includes three types of data: notes, note duration, and whether it is legato. Legato indicates that two notes need to be sung together.

[0114] S102: Determine the phonemes corresponding to the lyrics data based on the lyrics data contained in the song data.

[0115] S104: The musical score data and the phonemes are concatenated to obtain the first result.

[0116] First, the server determines the phonemes corresponding to the lyric data in the song data using a pinyin library. Then, the score data and the determined phonemes are concatenated to obtain a first result. Here, the pinyin library is a library for converting Chinese characters to pinyin, such as the PyPinyin library in Python. A phoneme is the initial or final sound corresponding to a lyric. For example, if the lyric is "我", according to the pinyin library, the phonemes corresponding to the lyric "我" are w and o.

[0117] When concatenating the score data and the phonemes to obtain the first result, the phonemes and the score data can be concatenated according to the time dimension, that is, the time point where the phoneme is located is determined in the score data, and according to the determined time point, the phoneme and the score data are concatenated. For example, according to the score data, it is determined that phoneme 1 is sung at the 10th second of the score. The 10th second can be used as the time point of phoneme 1, and according to this time point, phoneme 1 is concatenated with the score data, and the concatenated result is used as the first result.

[0118] S106: Input the first result into the melody prediction layer of the to-be-trained singing synthesis model to predict the first melody distribution of the song.

[0119] In this specification, the to-be-trained singing synthesis model at least includes a melody prediction layer and a singing prediction layer. The server inputs the first result into the melody prediction layer of the to-be-trained singing synthesis model to predict the first melody distribution of the song. Here, the melody distribution contains information about the phonemes corresponding to the lyric data. For each phoneme, different singing styles can be used to express the phoneme. Therefore, the melody distribution is a set of different singing styles used to express different phonemes, that is, the information of the phoneme is the singing style corresponding to the phoneme. For example, assume a song contains 2 phonemes, phoneme 1 has two singing styles A and B, and phoneme 2 has one singing style C. Then the melody distribution is a set containing two ways of expressing the song, namely AC (expressing phoneme 1 in the way of A and expressing phoneme 2 in the way of C) and BC (expressing phoneme 1 in the way of B and expressing phoneme 2 in the way of C).

[0120] Specifically, as Figure 2 shown, Figure 2This diagram illustrates the structure of a melody prediction layer provided in this specification. The melody prediction layer includes at least a feature extraction layer, a duration prediction layer, a fundamental frequency prediction layer, a Mel spectrum prediction layer, and a distribution prediction layer. First, the server inputs the first result into the feature extraction layer of the melody prediction layer to obtain the latent features of the phonemes. Then, the latent features are input into the duration prediction layer of the melody prediction layer to obtain the first pronunciation duration of the phonemes. Next, based on the first pronunciation duration of the phonemes, the latent features of the phonemes are expanded to obtain the second result. The second result is then input into the fundamental frequency prediction layer of the melody prediction layer to obtain the first fundamental frequency of the song's audio frames. Then, the second result and the first fundamental frequency of the song's audio frames are concatenated to obtain the third result. The third result is input into the Mel spectrum prediction layer of the melody prediction layer to obtain the second Mel spectrum of the song. The second Mel spectrum and the second result are concatenated to obtain the fourth result. Finally, the fourth result is input into the distribution prediction layer of the melody prediction layer to obtain the first melody distribution of the song.

[0121] The second result is obtained by expanding the implicit features of a phoneme based on its first pronunciation duration in the time dimension. The second result represents the pronunciation duration information of the phoneme. For example, assuming there is only one phoneme in a song, and the predicted pronunciation duration of this phoneme is 10 frames, this phoneme is one initial or final in the sound dimension, and 10 in the time dimension. Therefore, the implicit representation of this phoneme is expanded by 10 times, which means that the initial or final corresponding to this phoneme needs to be pronounced for 10 frames.

[0122] When concatenating the second result with the first fundamental frequency of the song's audio frame to obtain the third result, the server can concatenate the second result with the first fundamental frequency of the song's audio frame according to the time dimension, and use the concatenated result as the third result. Similarly, when concatenating the second Mel spectrum with the second result to obtain the fourth result, the server can also concatenate the second Mel spectrum with the second result according to the time dimension, and use the concatenated result as the fourth result. The specific process is similar to the process of obtaining the first result described above, and will not be repeated here.

[0123] S108: Based on the audio data, determine the first Mel spectrum of the song, and input the first Mel spectrum into the singing prediction layer of the singing synthesis model to be trained to obtain the predicted audio and the second melody distribution.

[0124] The server can determine the first melodic spectrum of a song based on the audio data, and input the first melodic spectrum into the vocal prediction layer of the vocal synthesis model to be trained, to obtain the predicted audio and the second melody distribution. Specifically, when determining the first melodic spectrum of a song based on the audio data, the server can use a preset first algorithm to determine the first melodic spectrum of the song. The first algorithm can be an algorithm included in the librosa toolkit, or any existing method for determining the melodic spectrum of a song based on audio data; this specification does not impose specific limitations.

[0125] Specifically, such as Figure 3 As shown, Figure 3 This is a schematic diagram of the structure of a singing prediction layer provided in this specification. The singing prediction layer may include a spectrogram distribution prediction layer, a conversion layer, and a differentiable digital signal processing layer. The spectrogram distribution prediction layer can be a network layer pre-trained separately based on the Mel spectrum and its corresponding spectrogram distribution, or it can be a network layer trained as part of the singing synthesis model. The conversion layer is used to align the melody distribution and the spectrogram distribution, realizing the conversion between them. The conversion layer can be a pre-trained network layer that reversibly converts the melody distribution and the spectrogram distribution, or it can be any existing network layer that achieves reversibility between the melody distribution and the spectrogram distribution, or it can be a network layer trained as part of the singing synthesis model. The differentiable digital signal processing layer is a network layer based on Differentiable Digital Signal Processing (DDSP) technology. The differentiable digital signal processing layer can be a pre-trained network layer, or it can be a network layer trained as part of the singing synthesis model; this specification does not specifically limit its use. For the sake of convenience, the spectral distribution prediction layer, conversion layer, and differentiable digital signal processing layer in this manual are all trained as part of the vocal synthesis model.

[0126] The server can determine the first melody spectrum of the song based on the audio data using a preset first algorithm. The first melody spectrum is then input into the spectral distribution prediction layer of the vocal prediction layer to obtain the predicted spectral distribution of the song. Next, the predicted spectral distribution of the song is input into the differentiable digital signal processing layer of the vocal prediction layer to obtain the predicted audio of the song. Finally, the predicted spectral distribution is input into the transformation layer of the vocal prediction layer to obtain the second melody distribution.

[0127] S110: Based on the difference between the first melody distribution and the second melody distribution, and the difference between the audio data and the predicted audio, determine the target loss, and train the singing synthesis model to be trained with the minimum target loss as the optimization objective.

[0128] The server can determine the target loss based on the differences between the first and second melody distributions, and the differences between the audio data and the predicted audio. It then trains the vocal synthesis model with the goal of minimizing this target loss. Specifically, the server determines the first loss based on the difference between the audio data and the predicted audio, and the second loss based on the difference between the first and second melody distributions. The sum of the first and second losses is then used as the target loss, and the vocal synthesis model is trained with the goal of minimizing this target loss.

[0129] As can be seen from the above method, this application trains the vocal synthesis model to be trained by using pre-collected song data when synthesizing vocal sounds. During training, the phonemes of the song are first determined based on the lyrics data. These phonemes are then concatenated with the musical score data, and the concatenation result is input into the melody prediction layer of the vocal synthesis model to obtain the first melody distribution. Next, based on the audio data, a predicted spectrogram distribution is obtained through the spectrogram distribution prediction layer of the vocal prediction layer. The predicted spectrogram distribution is then input into the conversion layer and the differentiable digital signal processing layer to obtain the second melody distribution and the predicted audio. Then, based on the predicted audio and the audio data, a first loss is determined, and a second loss is determined based on the first and second melody distributions. The sum of the first and second losses is used as the target loss, and the vocal synthesis model is trained based on the target loss. Obtaining the vocal sound through the differentiable digital signal processing layer of the vocal prediction layer improves the sampling rate of the vocal sound and reduces vibrato. Meanwhile, by predicting the Mel spectrum and fundamental frequency of the audio frames of the song, and then predicting the melody distribution of the song, the accuracy of the generated singing pronunciation is improved.

[0130] In step S110 above, when determining the target loss based on the difference between the first and second melody distributions and the difference between the audio data and the predicted audio, the server can determine the second pronunciation duration of a phoneme using a preset second algorithm based on the audio data. A third loss is determined based on the difference between the first and second pronunciation durations. Then, the target loss is determined based on the difference between the first and second melody distributions, the difference between the audio data and the predicted audio, and the third loss. The second algorithm can be any existing method for determining the pronunciation duration of a phoneme based on audio data. When determining the target loss based on the difference between the first and second melody distributions, the difference between the audio data and the predicted audio, and the third loss, the server can determine the first loss based on the predicted audio and the audio data, and determine the second loss based on the first and second melody distributions. The sum of the first, second, and third losses is then used as the target loss.

[0131] The server can also determine the second fundamental frequency of the song's audio frames using a preset third algorithm based on the audio data. A fourth loss is determined based on the difference between the first and second fundamental frequencies. Then, the target loss is determined based on the difference between the first and second melody distributions, the difference between the audio data and the predicted audio, and the fourth loss. The third algorithm can be an algorithm included in the ParseMouth toolkit, or any existing method for determining the fundamental frequency of a song's audio frames based on audio data; this specification does not specifically limit this method. When determining the target loss based on the difference between the first and second melody distributions, the difference between the audio data and the predicted audio, and the fourth loss, the server can determine the first loss based on the predicted audio and the audio data, and the second loss based on the first and second melody distributions. The sum of the first, second, and fourth losses is then used as the target loss.

[0132] The server can also determine the fifth loss based on the difference between the second and first Melodic spectra. Then, based on the difference between the first and second melody distributions, the difference between the audio data and the predicted audio, and the fifth loss, the target loss is determined. When determining the target loss based on the difference between the first and second melody distributions, the difference between the audio data and the predicted audio, and the fifth loss, the server can determine the first loss based on the predicted audio and audio data, and the fifth loss based on the first and second melody distributions. The sum of the first, second, and fifth losses is then used as the target loss.

[0133] Additionally, the server can use the sum of the first, second, third, fourth, and fifth losses as the target loss. Alternatively, the server can arbitrarily determine two losses from the third, fourth, and fifth losses, combine these determined losses with the first and second losses, and use the sum of the combined losses as the target loss. For example, if the determined losses are the third and fourth losses, then the target loss is the sum of the first, second, third, and fourth losses.

[0134] In step S108 above, when the predicted spectral distribution of the song is input into the differentiable digital signal processing layer of the vocal prediction layer to obtain the predicted audio of the song, the server can input the predicted spectral distribution of the song into the differentiable digital signal processing layer of the vocal prediction layer to determine the harmonic parameters and noise parameters of the audio frames of the song. Then, for each audio frame of the song, the predicted fundamental frequency corresponding to that audio frame is determined. Based on the predicted fundamental frequency and the harmonic parameters of that audio frame, the harmonic signal of that audio frame is determined, and based on the predicted fundamental frequency and the noise parameters of that audio frame, the noise signal of that audio frame is determined. Here, the harmonic parameters are parameters of the harmonic amplitude, and the noise parameters are parameters of the noise amplitude. The harmonic signal and noise signal of that audio frame can be determined using the following formula:

[0135]

[0136]

[0137] in, This represents a harmonic signal, where x represents the number of harmonic parameters. The parameter representing the amplitude of the k-th harmonic. Represents a sinusoidal signal. Represents phase information, This represents the noise signal, where y represents the number of noise parameters. The parameter representing the amplitude of the k-th noise wave, when The harmonic phase information can be calculated using the following formula:

[0138]

[0139] in, This is the predicted fundamental frequency corresponding to the audio frame. The audio sampling rate can be preset according to business needs; for example, it can be set to 48000. The value range is 1 to x.

[0140] when Represents noise phase information. It is a random number. The value range is 1 to y.

[0141] Next, the sum of the harmonic signal and the noise signal of the audio frame is determined as the audio signal of the audio frame. Then, based on the audio signals of each audio frame, the predicted audio of the song is determined.

[0142] When the predicted spectral distribution of a song is input into the differentiable digital signal processing layer of the vocal prediction layer to determine the harmonic parameters and noise parameters of the song's audio frames, the server can input the predicted spectral distribution of the song into the differentiable digital signal processing layer of the vocal prediction layer to determine a specified number of harmonic parameters and a specified number of noise parameters of the song's audio frames.

[0143] In this specification, the song has multiple audio frames, and each audio frame corresponds to a fundamental frequency. Therefore, after inputting the second result into the fundamental frequency prediction layer in step S106 above, the first fundamental frequency obtained is the first fundamental frequency of each audio frame.

[0144] After the vocal synthesis model to be trained in step S110 is completed, the server can respond to the user's input operation, determine the lyrics data and sheet music data of the song to be synthesized, and then determine the phonemes corresponding to the user-input lyrics data as the phonemes to be synthesized. The phonemes to be synthesized are then concatenated with the user-input sheet music data to obtain a first result. Next, the first result is input into the melody prediction layer of the trained vocal synthesis model to predict the melody distribution of the song to be synthesized. Then, the melody distribution of the song to be synthesized is input into the transformation layer in the vocal prediction layer of the trained vocal synthesis model to obtain the spectrogram distribution of the song to be synthesized. Finally, the spectrogram distribution of the song to be synthesized is input into the differentiable digital signal processing layer in the vocal prediction layer of the trained vocal synthesis model to obtain the audio of the song to be synthesized.

[0145] The server can also respond to user input, determining the lyrics and sheet music data of the song to be synthesized, and identifying the phonemes corresponding to the user-input lyrics as the phonemes to be synthesized. The phonemes to be synthesized are then concatenated with the user-input sheet music data. The concatenated result is input into the feature extraction layer of the melody prediction layer of the trained vocal synthesis model to obtain the latent features of the phonemes to be synthesized. These latent features are then input into the duration prediction layer to obtain the pronunciation duration of the phonemes. Based on the pronunciation duration, the latent features of the phonemes to be synthesized are expanded to obtain an expanded result. This expanded result is then input into the fundamental frequency prediction layer to obtain the fundamental frequency of the audio frames of the song to be synthesized. The expanded result and the fundamental frequency are then concatenated to obtain a first concatenation result, which is then input into the Mel spectrum prediction layer to obtain the Mel spectrum of the song to be synthesized. Finally, the Mel spectrum and the expanded result are concatenated to obtain a second concatenation result, which is then input into the distribution prediction layer to obtain the melody distribution of the song to be synthesized. Then, the melody distribution is input into the transformation layer of the vocal prediction layer of the trained vocal synthesis model to obtain the spectrogram distribution of the song to be synthesized. The spectrogram distribution is then input into the differentiable digital signal processing layer to obtain the audio of the song to be synthesized.

[0146] Among them, the data whose names are inconsistent with the first result in the above training process, the second result in the above training process, the third result in the above training process, and the fourth result in the above training process after the first splicing result are all data of the same type, only their names are different in the training and application process.

[0147] In this specification, the overall result of the singing voice synthesis model is as follows: Figure 4 As shown, Figure 4 This is a schematic diagram of the structure of a singing voice synthesis model provided in this specification. The singing voice synthesis model includes a melody prediction layer and a singing voice prediction layer. The melody prediction layer includes a feature extraction layer, a duration prediction layer, a fundamental frequency prediction layer, a Mel spectrum prediction layer, and a distribution prediction layer. The singing voice prediction layer includes a spectral distribution prediction layer, a conversion layer, and a differentiable digital signal processing layer.

[0148] In this specification, the server can train the entire singing voice synthesis model or train individual network layers within the model. For example, after determining the third loss, the duration prediction module of the singing voice synthesis model can be trained separately based on this loss. Similarly, after determining the first loss, the spectrogram prediction layer of the singing voice synthesis model can be trained, or the spectrogram prediction layer and the differentiable digital signal processing layer can be trained together. Likewise, other network layers of the singing voice synthesis model can be trained individually in the same manner, which will not be elaborated upon further here.

[0149] In this specification, the melody prediction layer and the singing prediction layer of the singing synthesis model, or the feature extraction layer, duration prediction layer, fundamental frequency prediction layer, melodic spectrum prediction layer and distribution prediction layer contained in the melody prediction layer, and the spectral distribution prediction layer, transformation layer and differentiable digital signal processing layer contained in the singing prediction layer, the structure of the above network layers can be set according to business needs. For example, the feature extraction layer in the singing synthesis model can be a network layer with a depth of 6 TransformerEncoder, and the duration prediction layer can be set as a 3-layer one-dimensional convolutional network layer.

[0150] In this specification, the server can input the melody distribution of a song into the vocal prediction layer of the vocal synthesis model to be trained, obtain the predicted audio of the song, and determine the target loss based on the difference between the audio data and the predicted audio. Then, the vocal synthesis model to be trained is trained based on the target loss.

[0151] The above describes one or more implementations of this specification. Based on the same idea, this specification also provides a training device for a corresponding singing voice synthesis model. The singing voice synthesis model to be trained includes a melody prediction layer and a singing voice prediction layer, such as... Figure 5 As shown.

[0152] Figure 5 A schematic diagram of the structure of a training device for a singing voice synthesis model provided in this specification includes:

[0153] The acquisition module 200 is used to acquire song data of pre-collected songs, wherein the song data includes at least lyrics data, sheet music data and audio data;

[0154] Phoneme module 202 is used to determine the phonemes corresponding to the lyrics data based on the lyrics data contained in the song data;

[0155] The determining module 204 is used to concatenate the musical score data and the phonemes to obtain a first result;

[0156] Melody module 206 is used to input the first result into the melody prediction layer of the singing synthesis model to be trained, and predict the predicted melody distribution of the song, wherein the predicted melody distribution includes information on the phonemes corresponding to the lyrics data;

[0157] The audio module 208 is used to determine the first melodic spectrum of the song based on the audio data, and input the first melodic spectrum into the singing prediction layer of the singing synthesis model to be trained to obtain the predicted audio and the second melody distribution.

[0158] Training module 210 determines the target loss based on the difference between the first melody distribution and the second melody distribution, and the difference between the audio data and the predicted audio, and trains the singing synthesis model to be trained with the minimum target loss as the optimization objective.

[0159] Optionally, the melody prediction layer includes a feature extraction layer, a duration prediction layer, a fundamental frequency prediction layer, a Mel spectrum prediction layer, and a distribution prediction layer;

[0160] The melody module 206 is specifically used to: input the first result into the feature extraction layer of the melody prediction layer to obtain the implicit features of the phoneme; input the implicit features into the duration prediction layer of the melody prediction layer to obtain the first pronunciation duration of the phoneme; expand the implicit features of the phoneme according to the predicted pronunciation duration of the phoneme to obtain a second result; input the second result into the fundamental frequency prediction layer of the melody prediction layer to obtain the first fundamental frequency of the audio frame of the song; concatenate the second result and the first fundamental frequency of the audio frame of the song to obtain a third result; input the third result into the Mel spectrum prediction layer of the melody prediction layer to obtain the second Mel spectrum of the song; concatenate the second Mel spectrum and the second result to obtain a fourth result; and input the fourth result into the distribution prediction layer of the melody prediction layer to obtain the first melody distribution of the song.

[0161] Optionally, the singing prediction layer includes a spectrogram distribution prediction layer, a conversion layer, and a differentiable digital signal processing layer;

[0162] The audio module 208 is specifically used to: determine the first Mel spectrum of the song based on the audio data using a preset first algorithm; input the first Mel spectrum into the spectral distribution prediction layer of the singing prediction layer to obtain the predicted spectral distribution of the song; input the predicted spectral distribution of the song into the differentiable digital signal processing layer of the singing prediction layer to obtain the predicted audio of the song; and input the predicted spectral distribution into the conversion layer of the singing prediction layer to obtain the second melody distribution.

[0163] Optionally, the training module 210 is specifically configured to determine a first loss based on the difference between the audio data and the predicted audio, and to determine a second loss based on the difference between the first melody distribution and the second melody distribution; and to use the sum of the first loss and the second loss as the target loss.

[0164] Optionally, the training module 210 is specifically configured to: determine the second pronunciation duration of the phoneme based on the audio data using a preset second algorithm; determine a third loss based on the difference between the first pronunciation duration and the second pronunciation duration; and determine the target loss based on the difference between the first melody distribution and the second melody distribution, the difference between the audio data and the predicted audio, and the third loss.

[0165] Optionally, the training module 210 is specifically configured to: determine the second fundamental frequency of the audio frame of the song using a preset third algorithm based on the audio data; determine a fourth loss based on the difference between the first fundamental frequency and the second fundamental frequency; and determine the target loss based on the difference between the first melody distribution and the second melody distribution, the difference between the audio data and the predicted audio, and the fourth loss.

[0166] Optionally, the training module 210 is specifically configured to: determine a fifth loss based on the difference between the second Mel spectrum and the first Mel spectrum; and determine the target loss based on the difference between the first melody distribution and the second melody distribution, the difference between the audio data and the predicted audio, and the fifth loss.

[0167] Optionally, the training module 210 is specifically configured to: determine the second pronunciation duration of the phoneme and the second fundamental frequency of the audio frame of the song based on the audio data using a preset algorithm; determine a third loss based on the difference between the first pronunciation duration and the second pronunciation duration; determine a fourth loss based on the difference between the first fundamental frequency and the second fundamental frequency; determine a fifth loss based on the difference between the second Mel spectrum and the first Mel spectrum; and determine a target loss based on the difference between the first melody distribution and the second melody distribution, the difference between the audio data and the predicted audio, the third loss, the fourth loss, and the fifth loss.

[0168] Optionally, the audio module 208 is specifically configured to: input the predicted spectral distribution of the song into the differentiable digital signal processing layer of the vocal prediction layer to determine the harmonic parameters and noise parameters of the audio frames of the song; for each audio frame of the song, determine the predicted fundamental frequency corresponding to that audio frame; determine the harmonic signal of the audio frame based on the predicted fundamental frequency and the harmonic parameters of that audio frame, and determine the noise signal of the audio frame based on the predicted fundamental frequency and the noise parameters of that audio frame; determine the sum of the harmonic signal and the noise signal of the audio frame as the audio signal of that audio frame; and determine the predicted audio of the song based on the audio signals of each audio frame.

[0169] Optionally, the device further includes:

[0170] Application module 212 is used to respond to user input operations by determining the lyrics data and sheet music data of the song to be synthesized input by the user; determining the phonemes corresponding to the lyrics data input by the user as the phonemes to be synthesized; concatenating the phonemes to be synthesized with the sheet music data input by the user to obtain a first result; inputting the first result into the melody prediction layer of the trained vocal synthesis model to predict the melody distribution of the song to be synthesized; inputting the melody distribution of the song to be synthesized into the conversion layer in the vocal prediction layer of the trained vocal synthesis model to obtain the spectral distribution of the song to be synthesized; and inputting the spectral distribution of the song to be synthesized into the differentiable digital signal processing layer in the vocal prediction layer of the trained vocal synthesis model to obtain the audio of the song to be synthesized.

[0171] Optionally, the melody prediction layer includes a feature extraction layer, a duration prediction layer, a fundamental frequency prediction layer, a Mel spectrum prediction layer, and a distribution prediction layer;

[0172] Application module 212 is used to respond to user input operations by determining the lyrics data and sheet music data of the song to be synthesized input by the user; determining the phoneme corresponding to the lyrics data input by the user as the phoneme to be synthesized; concatenating the phoneme to be synthesized with the sheet music data input by the user; inputting the concatenated result into the feature extraction layer of the melody prediction layer of the trained vocal synthesis model to obtain the implicit features of the phoneme to be synthesized; inputting the implicit features of the phoneme to be synthesized into the duration prediction layer to obtain the pronunciation duration of the phoneme to be synthesized; expanding the implicit features of the phoneme to be synthesized according to the pronunciation duration of the phoneme to be synthesized to obtain the expanded result; and inputting the expanded result into the feature extraction layer of the melody prediction layer of the trained vocal synthesis model to obtain the implicit features of the phoneme to be synthesized. The augmentation result is input into the fundamental frequency prediction layer to obtain the fundamental frequency of the audio frame of the song to be synthesized; the augmentation result and the fundamental frequency are concatenated to obtain a first concatenation result; the first concatenation result is input into the Mel spectrum prediction layer to obtain the Mel spectrum of the song to be synthesized; the Mel spectrum and the augmentation result are concatenated to obtain a second concatenation result; the second concatenation result is input into the distribution prediction layer to obtain the melody distribution of the song to be synthesized; the melody distribution is input into the transformation layer of the vocal prediction layer of the trained vocal synthesis model to obtain the spectral distribution of the song to be synthesized; the spectral distribution is input into the differentiable digital signal processing layer to obtain the audio of the song to be synthesized.

[0173] This specification also provides a computer-readable storage medium storing a computer program that can be used to execute the above-described... Figure 1 A training method for a singing voice synthesis model is provided.

[0174] This instruction manual also provides Figure 6 One of the corresponding Figure 1 A schematic diagram of the structure of an electronic device. (e.g.) Figure 6 As shown, at the hardware level, this electronic device includes a processor, internal bus, network interface, memory, and non-volatile memory, and may also include other hardware required for business operations. The processor reads the corresponding computer program from the non-volatile memory into memory and then runs it to achieve the above. Figure 1 The training method for the singing voice synthesis model.

[0175] Of course, in addition to software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of hardware and software. In other words, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0176] In the 1990s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the methodology). However, with technological advancements, many methodological improvements today can be considered direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that a methodological improvement cannot be implemented using hardware physical modules. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program and "integrate" a digital system onto a PLD themselves, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must also be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also understand that by simply performing some logic programming on the method flow using one of these hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.

[0177] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0178] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0179] For ease of description, the above devices are described in terms of function, divided into various units. Of course, in implementing this specification, the functions of each unit can be implemented in one or more software and / or hardware components.

[0180] Those skilled in the art will understand that embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this specification may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0181] This specification is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0182] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0183] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0184] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0185] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0186] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0187] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0188] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this specification may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0189] This specification can be described in the general context of computer-executable instructions that are executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This specification can also be practiced in distributed computing environments, where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0190] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0191] The above description is merely an embodiment of this specification and is not intended to limit this specification. Various modifications and variations can be made to this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of the claims of this specification.

Claims

1. A training method for a singing voice synthesis model, characterized in that, The song synthesis model to be trained comprises a melody prediction layer and a song prediction layer; the method comprises: acquiring song data of a song collected in advance, wherein the song data at least comprises lyric data, score data and audio data; determining phonemes corresponding to the lyric data according to the lyric data contained in the song data; splicing the score data and the phonemes to obtain a first result; inputting the first result into the melody prediction layer of the song synthesis model to be trained to predict a first melody distribution of the song, wherein the first melody distribution contains information of the phonemes corresponding to the lyric data; determining a first mel-spectrogram of the song according to the audio data, inputting the first mel-spectrogram into the song prediction layer of the song synthesis model to be trained to obtain predicted audio and a second melody distribution; determining a target loss according to the difference between the first melody distribution and the second melody distribution and the difference between the audio data and the predicted audio, and taking the minimum of the target loss as an optimization target to train the song synthesis model to be trained.

2. The method of claim 1, wherein, The melody prediction layer comprises a feature extraction layer, a duration prediction layer, a fundamental frequency prediction layer, a mel-spectrogram prediction layer and a distribution prediction layer; inputting the first result into the melody prediction layer of the song synthesis model to be trained to obtain the first melody distribution of the song, specifically comprising: inputting the first result into the feature extraction layer of the melody prediction layer to obtain hidden features of the phonemes; inputting the hidden features into the duration prediction layer of the melody prediction layer to obtain a first pronunciation duration of the phonemes; performing extension processing on the hidden features of the phonemes according to the first pronunciation duration of the phonemes to obtain a second result; inputting the second result into the fundamental frequency prediction layer of the melody prediction layer to obtain a first fundamental frequency of an audio frame of the song; splicing the second result and the first fundamental frequency of the audio frame of the song to obtain a third result; inputting the third result into the mel-spectrogram prediction layer of the melody prediction layer to obtain a second mel-spectrogram of the song; splicing the second mel-spectrogram and the second result to obtain a fourth result; inputting the fourth result into the distribution prediction layer of the melody prediction layer to obtain the first melody distribution of the song.

3. The method of claim 1 or 2, wherein, The song prediction layer comprises a sound spectrum distribution prediction layer, a conversion layer and a differentiable digital signal processing layer; determining a first mel-spectrogram of the song according to the audio data, inputting the first mel-spectrogram into the song prediction layer of the song synthesis model to be trained to obtain predicted audio and a second melody distribution, specifically comprising: determining the first mel-spectrogram of the song by using a preset first algorithm according to the audio data; inputting the first mel-spectrogram into the sound spectrum distribution prediction layer of the song prediction layer to obtain a predicted sound spectrum distribution of the song; inputting the predicted sound spectrum distribution of the song into the differentiable digital signal processing layer of the song prediction layer to obtain the predicted audio of the song, and inputting the predicted sound spectrum distribution into the conversion layer of the song prediction layer to obtain the second melody distribution.

4. The method of claim 3, wherein, According to the difference between the first melody distribution and the second melody distribution, the difference between the audio data and the predicted audio, a target loss is determined, specifically comprising: According to the difference between the audio data and the predicted audio, a first loss is determined, and according to the difference between the first melody distribution and the second melody distribution, a second loss is determined; The sum of the first loss and the second loss is taken as the target loss.

5. The method of claim 2, wherein, According to the difference between the first melody distribution and the second melody distribution, the difference between the audio data and the predicted audio, a target loss is determined, specifically comprising: According to the audio data, a second pronunciation duration of the phoneme is determined by using a preset second algorithm; According to the difference between the first pronunciation duration and the second pronunciation duration, a third loss is determined; According to the difference between the first melody distribution and the second melody distribution, the difference between the audio data and the predicted audio, and the third loss, the target loss is determined.

6. The method of claim 2, wherein, According to the difference between the first melody distribution and the second melody distribution, the difference between the audio data and the predicted audio, a target loss is determined, specifically comprising: According to the audio data, a second fundamental frequency of the audio frame of the song is determined by using a preset third algorithm; According to the difference between the first fundamental frequency and the second fundamental frequency, a fourth loss is determined; According to the difference between the first melody distribution and the second melody distribution, the difference between the audio data and the predicted audio, and the fourth loss, the target loss is determined.

7. The method of claim 2, wherein, According to the difference between the first melody distribution and the second melody distribution, the difference between the audio data and the predicted audio, a target loss is determined, specifically comprising: According to the difference between the second mel spectrum and the first mel spectrum, a fifth loss is determined; According to the difference between the first melody distribution and the second melody distribution, the difference between the audio data and the predicted audio, and the fifth loss, the target loss is determined.

8. The method of claim 2, wherein, According to the difference between the first melody distribution and the second melody distribution, the difference between the audio data and the predicted audio, a target loss is determined, specifically comprising: According to the audio data, a second pronunciation duration of the phoneme and a second fundamental frequency of the audio frame of the song are determined by using a preset algorithm; According to the difference between the first pronunciation duration and the second pronunciation duration, a third loss is determined; according to the difference between the first fundamental frequency and the second fundamental frequency, a fourth loss is determined; according to the difference between the second mel spectrum and the first mel spectrum, a fifth loss is determined; According to the difference between the first melody distribution and the second melody distribution, the difference between the audio data and the predicted audio, the third loss, the fourth loss and the fifth loss, the target loss is determined.

9. The method of claim 4, wherein, The predicted sound spectrum distribution of the song is input into the differentiable digital signal processing layer of the song prediction layer to obtain the predicted audio of the song, specifically comprising: The predicted sound spectrum distribution of the song is input into the differentiable digital signal processing layer of the song prediction layer to determine the harmonic parameters and noise parameters of the audio frame of the song; For each audio frame of the song, a predicted fundamental frequency corresponding to the audio frame is determined; According to the predicted fundamental frequency corresponding to the audio frame and the harmonic parameter of the audio frame, a harmonic signal of the audio frame is determined, and according to the predicted fundamental frequency corresponding to the audio frame and the noise parameter of the audio frame, a noise signal of the audio frame is determined; The sum of the harmonic signal of the audio frame and the noise signal of the audio frame is determined as the audio signal of the audio frame; According to the audio signals of the audio frames, a predicted audio of the song is determined.

10. The method of claim 3, wherein, The method further comprises: In response to a user's input operation, determining lyrics data and score data of a to-be-synthesized song input by the user; Determining phonemes corresponding to the lyrics data input by the user as to-be-synthesized phonemes; Splicing the to-be-synthesized phonemes with the score data input by the user to obtain a first result; Inputting the first result into a melody prediction layer of a trained singing voice synthesis model to predict a melody distribution of the to-be-synthesized song; Inputting the melody distribution of the to-be-synthesized song into a conversion layer in a singing voice prediction layer of the trained singing voice synthesis model to obtain a spectrogram distribution of the to-be-synthesized song; Inputting the spectrogram distribution of the to-be-synthesized song into a differentiable digital signal processing layer in the singing voice prediction layer of the trained singing voice synthesis model to obtain an audio of the to-be-synthesized song.

11. The method of claim 3, wherein, The melody prediction layer comprises a feature extraction layer, a duration prediction layer, a fundamental frequency prediction layer, a mel-spectrogram prediction layer, and a distribution prediction layer; The method further comprises: In response to a user's input operation, determining lyrics data and score data of a to-be-synthesized song input by the user; Determining phonemes corresponding to the lyrics data input by the user as to-be-synthesized phonemes; Splicing the to-be-synthesized phonemes with the score data input by the user; Inputting the spliced result into a feature extraction layer of a melody prediction layer of a trained singing voice synthesis model to obtain hidden features of the to-be-synthesized phonemes; Inputting the hidden features of the to-be-synthesized phonemes into the duration prediction layer to obtain pronunciation duration of the to-be-synthesized phonemes; According to the pronunciation duration of the to-be-synthesized phonemes, performing expansion processing on the hidden features of the to-be-synthesized phonemes to obtain an expansion result; Inputting the expansion result into the fundamental frequency prediction layer to obtain a fundamental frequency of an audio frame of the to-be-synthesized song; Splicing the expansion result and the fundamental frequency to obtain a first splicing result; Inputting the first splicing result into the mel-spectrogram prediction layer to obtain a mel-spectrogram of the to-be-synthesized song; Splicing the mel-spectrogram and the expansion result to obtain a second splicing result; Inputting the second splicing result into the distribution prediction layer to obtain a melody distribution of the to-be-synthesized song; Inputting the melody distribution into a conversion layer of a singing voice prediction layer of the trained singing voice synthesis model to obtain a spectrogram distribution of the to-be-synthesized song; Inputting the spectrogram distribution into the differentiable digital signal processing layer to obtain an audio of the to-be-synthesized song.

12. A device for training a singing voice synthesis model, characterized by, The to-be-trained singing voice synthesis model comprises a melody prediction layer and a singing voice prediction layer; and the device comprises: The acquisition module is configured to acquire song data of a pre-collected song, wherein the song data at least includes lyric data, score data, and audio data; The phoneme module is configured to determine phonemes corresponding to the lyric data according to the lyric data contained in the song data; The determination module is configured to splice the score data and the phonemes to obtain a first result; The melody module is configured to input the first result into a melody prediction layer of the to-be-trained singing synthesis model to predict a first melody distribution of the song, wherein the first melody distribution contains information of the phonemes corresponding to the lyric data; The audio module is configured to determine a first mel-spectrogram of the song according to the audio data, and input the first mel-spectrogram into a singing prediction layer of the to-be-trained singing synthesis model to obtain predicted audio and a second melody distribution; The training module is configured to determine a target loss according to a difference between the first melody distribution and the second melody distribution and a difference between the audio data and the predicted audio, and train the to-be-trained singing synthesis model with the target loss being minimized as an optimization target.

13. The apparatus of claim 12, wherein, The melody prediction layer includes a feature extraction layer, a duration prediction layer, a fundamental frequency prediction layer, a mel-spectrogram prediction layer, and a distribution prediction layer; The melody module is configured to input the first result into the feature extraction layer of the melody prediction layer to obtain hidden features of the phonemes, input the hidden features into the duration prediction layer of the melody prediction layer to obtain a first pronunciation duration of the phonemes, perform extension processing on the hidden features of the phonemes according to the predicted pronunciation duration of the phonemes to obtain a second result, and input the second result into the fundamental frequency prediction layer of the melody prediction layer to obtain a first fundamental frequency of an audio frame of the song. The second result and the first fundamental frequency of the audio frame of the song are spliced to obtain a third result, the third result is input into the mel-spectrogram prediction layer of the melody prediction layer to obtain a second mel-spectrogram of the song, the second mel-spectrogram and the second result are spliced to obtain a fourth result, and the fourth result is input into the distribution prediction layer of the melody prediction layer to obtain the first melody distribution of the song. The singing prediction layer includes a sound spectrum distribution prediction layer, a conversion layer, and a differentiable digital signal processing layer; 14. The apparatus of claim 12 or 13, wherein, The audio module is configured to determine the first mel-spectrogram of the song by using a preset first algorithm according to the audio data, input the first mel-spectrogram into the sound spectrum distribution prediction layer of the singing prediction layer to obtain a predicted sound spectrum distribution of the song, input the predicted sound spectrum distribution of the song into the differentiable digital signal processing layer of the singing prediction layer to obtain the predicted audio of the song, and input the predicted sound spectrum distribution into the conversion layer of the singing prediction layer to obtain the second melody distribution. The training module is configured to determine a first loss according to a difference between the audio data and the predicted audio, determine a second loss according to a difference between the first melody distribution and the second melody distribution, and take a sum of the first loss and the second loss as the target loss. ​ 15. The apparatus of claim 14, wherein, ​ 16. The apparatus of claim 13, wherein, The training module is specifically configured to determine a second pronunciation duration of the phoneme according to the audio data by using a preset second algorithm; determine a third loss according to a difference between the first pronunciation duration and the second pronunciation duration; and determine the target loss according to a difference between the first melody distribution and the second melody distribution, a difference between the audio data and the predicted audio, and the third loss.

17. The apparatus of claim 13, wherein, The training module is specifically configured to determine a second fundamental frequency of the audio frame of the song according to the audio data by using a preset third algorithm; determine a fourth loss according to a difference between the first fundamental frequency and the second fundamental frequency; and determine the target loss according to a difference between the first melody distribution and the second melody distribution, a difference between the audio data and the predicted audio, and the fourth loss.

18. The apparatus of claim 13, wherein, The training module is specifically configured to determine a fifth loss according to a difference between the second mel-spectrogram and the first mel-spectrogram; and determine the target loss according to a difference between the first melody distribution and the second melody distribution, a difference between the audio data and the predicted audio, and the fifth loss.

19. The apparatus of claim 13, wherein, The training module is specifically configured to determine a second pronunciation duration of the phoneme and a second fundamental frequency of the audio frame of the song according to the audio data by using a preset algorithm; determine a third loss according to a difference between the first pronunciation duration and the second pronunciation duration; determine a fourth loss according to a difference between the first fundamental frequency and the second fundamental frequency; determine a fifth loss according to a difference between the second mel-spectrogram and the first mel-spectrogram; and determine the target loss according to a difference between the first melody distribution and the second melody distribution, a difference between the audio data and the predicted audio, the third loss, the fourth loss, and the fifth loss.

20. The apparatus of claim 14, wherein, The audio module is specifically configured to input the predicted sound spectrum distribution of the song into a differentiable digital signal processing layer of the song prediction layer, determine a harmonic parameter and a noise parameter of an audio frame of the song; for each audio frame of the song, determine a predicted fundamental frequency corresponding to the audio frame; determine a harmonic signal of the audio frame according to the predicted fundamental frequency corresponding to the audio frame and the harmonic parameter of the audio frame, and determine a noise signal of the audio frame according to the predicted fundamental frequency corresponding to the audio frame and the noise parameter of the audio frame; determine a sum of the harmonic signal of the audio frame and the noise signal of the audio frame as an audio signal of the audio frame; and determine a predicted audio of the song according to the audio signals of the audio frames.

21. The apparatus of claim 14, wherein, The device further comprises: The application module is configured to, in response to an input operation of a user, determine lyric data and score data of a song to be synthesized input by the user, determine phonemes corresponding to the lyric data input by the user as phonemes to be synthesized, splice the phonemes to be synthesized with the score data input by the user to obtain a first result, input the first result into a melody prediction layer of a trained singing synthesis model to predict a melody distribution of the song to be synthesized, input the melody distribution of the song to be synthesized into a conversion layer in a singing prediction layer of the trained singing synthesis model to obtain a spectrum distribution of the song to be synthesized, and input the spectrum distribution of the song to be synthesized into a differentiable digital signal processing layer in the singing prediction layer of the trained singing synthesis model to obtain audio of the song to be synthesized.

22. The apparatus of claim 14, wherein, The melody prediction layer comprises a feature extraction layer, a duration prediction layer, a fundamental frequency prediction layer, a mel-spectrogram prediction layer, and a distribution prediction layer. The device further comprises: The application module is configured to, in response to an input operation of a user, determine lyric data and score data of a song to be synthesized input by the user, determine phonemes corresponding to the lyric data input by the user as phonemes to be synthesized, splice the phonemes to be synthesized with the score data input by the user, input a result after the splicing into a feature extraction layer of a melody prediction layer of a trained singing synthesis model to obtain hidden features of the phonemes to be synthesized, input the hidden features of the phonemes to be synthesized into the duration prediction layer to obtain pronunciation durations of the phonemes to be synthesized, perform expansion processing on the hidden features of the phonemes to be synthesized according to the pronunciation durations of the phonemes to be synthesized to obtain an expansion result, input the expansion result into the fundamental frequency prediction layer to obtain a fundamental frequency of an audio frame of the song to be synthesized, splice the expansion result and the fundamental frequency to obtain a first splicing result, input the first splicing result into the mel-spectrogram prediction layer to obtain a mel-spectrogram of the song to be synthesized, splice the mel-spectrogram and the expansion result to obtain a second splicing result, input the second splicing result into the distribution prediction layer to obtain a melody distribution of the song to be synthesized, input the melody distribution into a conversion layer of a singing prediction layer of the trained singing synthesis model to obtain a spectrum distribution of the song to be synthesized, and input the spectrum distribution into the differentiable digital signal processing layer to obtain audio of the song to be synthesized.

23. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, and the computer program, when executed by a processor, implements the method in any one of claims 1-11.

24. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein, The processor implements the method in any one of claims 1-11 when executing the program.

Images