Singing synthesis method, computer equipment and computer readable storage medium

Abstract

The invention relates to a singing synthesis method, computer equipment and a computer readable storage medium. The method comprises the steps of obtaining music score information of a to-be-synthesized song and to-be-input energy information; the music score information comprises a plurality of phonemes and pitches and phoneme durations corresponding to the phonemes; the to-be-input energy information is obtained through prediction based on music score information or obtained through extraction based on reference audio of a to-be-synthesized song; inputting the music score information and the to-be-input energy information into a pre-trained singing sound synthesis model, and obtaining a synthesized singing sound Mel spectrum through energy perception attention mechanism processing in the pre-trained singing sound synthesis model; the energy perception attention mechanism is used for combining the phoneme code of the music score information and the energy code of the to-be-input energy information to obtain a new energy code; and generating a synthesized song audio of the to-be-synthesized song according to the synthesized song Mel spectrum. By adopting the method, the stability of the model and the controllability of singing synthesis can be effectively improved, and the singing synthesis effect can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of singing voice synthesis, and in particular to a singing voice synthesis method, a computer device and a computer-readable storage medium. Background Art

[0002] With the development of Internet technology, singing synthesis technology has emerged, which can synthesize human-like singing by inputting music scores. Since there are many kinds of singing skills, the pronunciation of singing is an important skill, and the energy of audio is closely related to the pronunciation, so energy modeling is crucial in singing synthesis systems.

[0003] In traditional technologies, due to inaccurate energy extraction due to factors such as external noise and deviations in phoneme duration labeling, the model has low robustness and the singing synthesis effect is poor. Summary of the invention

[0004] Based on this, it is necessary to provide a singing synthesis method, a computer device and a computer-readable storage medium that can improve the robustness of the singing synthesis system in response to the above technical problems.

[0005] In a first aspect, the present application provides a singing synthesis method. The method comprises:

[0006] Obtaining music score information of a song to be synthesized and energy information to be input; the music score information includes a plurality of phonemes and the pitch and duration of each phoneme; the energy information to be input is predicted based on the music score information or extracted based on the reference audio of the song to be synthesized;

[0007] The music score information and the energy information to be input are input into a pre-trained singing synthesis model, and the synthesized singing mel spectrum is obtained by processing the energy-aware attention mechanism in the pre-trained singing synthesis model; the energy-aware attention mechanism is used to combine the phoneme encoding of the music score information and the energy encoding of the energy information to be input to obtain a new energy encoding;

[0008] According to the synthesized singing mel-spectrogram, the synthesized singing audio of the song to be synthesized is generated.

[0009] In one embodiment, the music score information and the energy information to be input are input into a pre-trained singing synthesis model, and the synthesized singing mel spectrum is obtained by processing the energy-aware attention mechanism in the pre-trained singing synthesis model, including:

[0010] Obtaining the phoneme code according to the plurality of phonemes and the pitch and phoneme duration corresponding to each of the phonemes;

[0011] Based on the phoneme code and the energy code of the energy information to be input, the new energy code is obtained by processing the energy-aware attention mechanism;

[0012] The synthesized singing mel spectrum is obtained according to the encoding fusion result of the phoneme encoding and the new energy encoding.

[0013] In one embodiment, the pre-trained singing synthesis model includes a text encoder, and the phoneme encoding is obtained according to the plurality of phonemes and the pitch and phoneme duration corresponding to each of the phonemes, including:

[0014] Inputting the plurality of phonemes and the pitch corresponding to each of the phonemes into the text encoder to obtain a phoneme encoding result;

[0015] Phoneme expansion is performed according to the phoneme encoding result and the phoneme duration corresponding to each of the phonemes to obtain the phoneme encoding.

[0016] In one embodiment, the energy encoding based on the phoneme encoding and the energy information to be input is processed by the energy-aware attention mechanism to obtain the new energy encoding, including:

[0017] The phoneme encoding is used as a query for the energy-aware attention mechanism, and the energy encoding is used as a key and value for the energy-aware attention mechanism. The new energy encoding is obtained by processing the energy-aware attention mechanism.

[0018] In one embodiment, the pre-trained singing synthesis model further includes a decoder, and the encoding fusion result according to the phoneme encoding and the new energy encoding is used to obtain the synthesized singing mel spectrum, including:

[0019] Fusing the phoneme code and the new energy code to obtain the code fusion result;

[0020] The encoding fusion result is processed by the decoder to obtain the synthesized singing mel spectrum.

[0021] In one embodiment, the pre-trained singing voice synthesis model further includes an energy predictor or an energy extractor, and the step of obtaining the energy information to be input includes:

[0022] By means of the energy predictor, energy information predicted based on the phoneme coding is used as the energy information to be input;

[0023] Alternatively, the energy extractor is used to extract energy information of the reference audio as the energy information to be input.

[0024] In one embodiment, the pre-trained singing synthesis model is trained by the following method:

[0025] Acquire training data; the training data includes a plurality of sample phonemes in a sample song, a sample pitch and a sample phoneme duration corresponding to each of the sample phonemes, sample energy information of the sample song, and a sample singing Mel spectrum; the sample singing Mel spectrum is used as a sample label corresponding to the sample song;

[0026] Input the plurality of sample phonemes, the sample pitch and the sample phoneme duration corresponding to each of the sample phonemes, and the sample energy information into the singing synthesis model to be trained to obtain a predicted singing mel spectrum;

[0027] Based on the loss between the predicted singing mel-spectrogram and the sample singing mel-spectrogram, the singing synthesis model to be trained is trained until a preset training end condition is met, thereby obtaining the pre-trained singing synthesis model.

[0028] In one embodiment, obtaining training data includes:

[0029] Acquire singing audio data and sample music score information of the sample song; the sample music score information includes the multiple sample phonemes and the sample pitch and sample phoneme duration corresponding to each of the sample phonemes;

[0030] The sample energy information and the sample singing mel spectrum are extracted based on the singing audio data of the sample song.

[0031] In a second aspect, the present application further provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0032] Obtaining music score information of a song to be synthesized and energy information to be input; the music score information includes a plurality of phonemes and the pitch and duration of each phoneme; the energy information to be input is predicted based on the music score information or extracted based on the reference audio of the song to be synthesized;

[0033] The music score information and the energy information to be input are input into a pre-trained singing synthesis model, and the synthesized singing mel spectrum is obtained by processing the energy-aware attention mechanism in the pre-trained singing synthesis model; the energy-aware attention mechanism is used to combine the phoneme encoding of the music score information and the energy encoding of the energy information to be input to obtain a new energy encoding;

[0034] According to the synthesized singing mel-spectrogram, the synthesized singing audio of the song to be synthesized is generated.

[0035] In a third aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0036] Obtaining music score information of a song to be synthesized and energy information to be input; the music score information includes a plurality of phonemes and the pitch and duration of each phoneme; the energy information to be input is predicted based on the music score information or extracted based on the reference audio of the song to be synthesized;

[0037] The music score information and the energy information to be input are input into a pre-trained singing synthesis model, and the synthesized singing mel spectrum is obtained by processing the energy-aware attention mechanism in the pre-trained singing synthesis model; the energy-aware attention mechanism is used to combine the phoneme encoding of the music score information and the energy encoding of the energy information to be input to obtain a new energy encoding;

[0038] According to the synthesized singing mel-spectrogram, the synthesized singing audio of the song to be synthesized is generated.

[0039] In a fourth aspect, the present application further provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0040] Obtaining music score information of a song to be synthesized and energy information to be input; the music score information includes a plurality of phonemes and the pitch and duration of each phoneme; the energy information to be input is predicted based on the music score information or extracted based on the reference audio of the song to be synthesized;

[0041] The music score information and the energy information to be input are input into a pre-trained singing synthesis model, and the synthesized singing mel spectrum is obtained by processing the energy-aware attention mechanism in the pre-trained singing synthesis model; the energy-aware attention mechanism is used to combine the phoneme encoding of the music score information and the energy encoding of the energy information to be input to obtain a new energy encoding;

[0042] According to the synthesized singing mel-spectrogram, the synthesized singing audio of the song to be synthesized is generated.

[0043] The above-mentioned singing synthesis method, computer device and computer-readable storage medium obtain the score information of the song to be synthesized and the energy information to be input, wherein the score information includes multiple phonemes and the pitch and phoneme duration corresponding to each phoneme, and the energy information to be input is predicted based on the score information or extracted based on the reference audio of the song to be synthesized, and then the score information and the energy information to be input are input into the pre-trained singing synthesis model, and processed by the energy-aware attention mechanism in the pre-trained singing synthesis model to obtain the synthesized singing mel-spectrogram, and the energy-aware attention mechanism is used to combine the phoneme encoding of the score information and the energy encoding of the energy information to be input to obtain a new energy encoding, and then generate the synthesized singing audio of the song to be synthesized according to the synthesized singing mel-spectrogram, thereby realizing the optimization of the singing synthesis processing, and processing the phoneme encoding of the score information and the energy encoding of the energy information to be input based on the energy-aware attention mechanism can effectively improve the stability of the model, and can flexibly input energy information, thereby improving the controllability of singing synthesis and helping to improve the singing synthesis effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] FIG. 1 is a schematic flow diagram of a singing voice synthesis method in one embodiment;

[0045] FIG2 is a schematic diagram of a singing voice synthesis processing application architecture in one embodiment;

[0046] FIG3 is a schematic diagram of a singing synthesis processing training framework according to an embodiment;

[0047] FIG4 is a schematic flow diagram of a singing voice synthesis method in another embodiment;

[0048] FIG5 is a block diagram of a singing voice synthesis method device according to an embodiment;

[0049] FIG. 6 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0051] In one embodiment, as shown in FIG1 , a singing voice synthesis method is provided. This embodiment is illustrated by applying the method to a terminal. It is understandable that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. The server can be implemented as an independent server or a server cluster composed of multiple servers; the terminal can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, portable wearable devices, etc.

[0052] In this embodiment, the method includes the following steps S101 to S103. Among them:

[0053] In step S101, the music score information of the song to be synthesized and the energy information to be input are obtained;

[0054] The score information may include a plurality of phonemes and the pitch and duration of each phoneme.

[0055] As an example, the energy information to be input may be predicted based on music score information or extracted based on reference audio of the song to be synthesized.

[0056] In practical applications, singing synthesis processing can be performed based on the energy-aware singing synthesis system. For example, based on the singing synthesis function in the APP or client, SVS (Singing-voice-synthesis) technology can be used to obtain the music score information of the song to be synthesized, which may include but is not limited to phonemes, pitches, and durations, that is, multiple phonemes and the pitches and durations corresponding to each phoneme, combined with the energy information to be input, and then the singing of the specified music score content can be output.

[0057] In one example, for singing audio, the energy of the audio is related to the volume of the audio sound. The greater the energy, the louder the sound. The energy of the audio can be extracted from the audio waveform file as a feature of the audio, namely, energy information.

[0058] For example, when a user needs to synthesize singing voices, the singing voice synthesis function in the APP or client can be used to perform singing voice synthesis processing, the score information of the song to be synthesized can be input, and the reference audio input can be provided, which can be used to extract energy information based on the reference audio, so that the energy of the output singing voice and the energy of the reference audio have a certain similarity; the energy information to be input can also be obtained by predicting energy, that is, the energy information to be input can be obtained by predicting based on the score information or extracting based on the reference audio of the song to be synthesized. This makes the energy input flexible and can also be modified manually, which can improve the controllability of singing voice synthesis.

[0059] In step S102, the music score information and the energy information to be input are input into a pre-trained singing synthesis model, and the synthesized singing mel spectrum is obtained through the energy-aware attention mechanism in the pre-trained singing synthesis model;

[0060] Among them, the energy-aware attention mechanism can be used to combine the phoneme encoding of the music score information and the energy encoding of the energy information to be input to obtain a new energy encoding, which is obtained after aligning the time relationship between energy and phonemes.

[0061] In the process of singing synthesis, as shown in Figure 2, the phoneme string, pitch, phoneme duration (i.e., multiple phonemes and the pitch and duration corresponding to each phoneme), and energy can be used as input, that is, the score information and the energy information to be input are input into the pre-trained singing synthesis model, and the phoneme encoding can be obtained according to the multiple phonemes and the pitch and duration corresponding to each phoneme. Then, based on the phoneme encoding and the energy encoding of the energy information to be input, a new energy encoding can be obtained through the energy-aware attention mechanism, and then the corresponding singing mel spectrum (i.e., synthesized singing mel spectrum) can be synthesized according to the encoding fusion result of the phoneme encoding and the new energy encoding. Therefore, based on the combination of energy and phonemes through the attention mechanism, the robustness and control ability of the system for energy modeling are improved.

[0062] In step S103, the synthesized singing audio of the song to be synthesized is generated according to the synthesized singing mel-spectrogram.

[0063] After obtaining the synthesized singing mel-spectrogram, the synthesized singing mel-spectrogram can be processed by a separately trained vocoder to obtain the corresponding singing audio, that is, the synthesized singing audio of the song to be synthesized.

[0064] Compared with traditional methods, such as speech synthesis based on energy modeling, which directly adds energy to phoneme coding, the energy may be extracted inaccurately due to external noise and other reasons, and the phoneme duration annotation may also be biased, which will lead to low model robustness; or traditional singing synthesis systems that do not model energy, because the system has no energy input, will cause the model to have low control over energy, and the synthesized singing energy changes relatively smoothly and randomly. The technical solution of this embodiment, by adopting a highly energy-robust and controllable singing synthesis system for singing synthesis processing, can improve the robustness of the system based on the energy-aware attention mechanism; and provides an entry for energy input, which can not only use the energy automatically predicted by the model, but also the energy of the external input reference audio, and there is no need to limit the duration, content, etc. of the reference audio, while improving the controllability of the model energy.

[0065] In the above-mentioned singing synthesis method, the score information of the song to be synthesized and the energy information to be input are obtained, and then the score information and the energy information to be input are input into the pre-trained singing synthesis model. The synthesized singing mel-spectrogram is obtained through the energy-aware attention mechanism in the pre-trained singing synthesis model, and then the synthesized singing audio of the song to be synthesized is generated according to the synthesized singing mel-spectrogram, thereby realizing the optimization of the singing synthesis processing. The phoneme encoding of the score information and the energy encoding of the energy information to be input are processed based on the energy-aware attention mechanism, which can effectively improve the stability of the model, and can flexibly input energy information, thereby improving the controllability of singing synthesis and helping to improve the singing synthesis effect.

[0066] In one embodiment, in step S102, the music score information and the energy information to be input are input into a pre-trained singing synthesis model, and the synthesized singing mel spectrum is obtained by processing the energy-aware attention mechanism in the pre-trained singing synthesis model, which may include the following steps:

[0067] According to multiple phonemes and the pitch and duration corresponding to each phoneme, a phoneme code is obtained; based on the phoneme code and the energy code of the energy information to be input, a new energy code is obtained through processing by an energy-aware attention mechanism; according to the coding fusion result of the phoneme code and the new energy code, a synthesized singing mel-spectrogram is obtained.

[0068] In a specific implementation, the pre-trained singing synthesis model architecture in the energy-aware singing synthesis system shown in Figure 2 may include a text encoder, an energy-aware attention mechanism, and a decoder; in the singing synthesis process, the music score information (including multiple phonemes and the pitch and phoneme duration corresponding to each phoneme, such as the phoneme string, pitch, and phoneme duration in Figure 2) may be input into the pre-trained singing synthesis model, and the phoneme code may be obtained through the text encoder and phoneme expansion, and then the phoneme code and the predicted energy or the energy extracted from the reference audio (i.e., the energy information to be input) may be input into the energy-aware attention mechanism together to obtain a new energy code, and then the new energy code may be added to the phoneme code, and after being processed by the decoder, the corresponding singing mel-spectrogram (i.e., synthesized singing mel-spectrogram) may be synthesized.

[0069] In this embodiment, a phoneme code is obtained based on multiple phonemes and the pitch and duration corresponding to each phoneme, and then a new energy code is obtained based on the phoneme code and the energy code of the energy information to be input through an energy-aware attention mechanism. Then, based on the coding fusion result of the phoneme code and the new energy code, a synthesized singing mel-spectrogram is obtained, which can effectively improve the stability of the model and help improve the singing synthesis effect.

[0070] In one embodiment, the pre-trained singing synthesis model may include a text encoder, and obtaining a phoneme encoding according to a plurality of phonemes and the pitch and phoneme duration corresponding to each phoneme may include the following steps:

[0071] Multiple phonemes and the pitches corresponding to each phoneme are input into a text encoder to obtain a phoneme encoding result; phoneme expansion is performed according to the phoneme encoding result and the phoneme duration corresponding to each phoneme to obtain a phoneme encoding.

[0072] In one example, the text encoder in the pre-trained singing synthesis model can be constructed based on the corresponding module reference in the Fastspeech speech synthesis system, or other acoustic models can be used, which is not specifically limited in this embodiment. As shown in Figure 2, the phoneme string and pitch (i.e., multiple phonemes and the pitch corresponding to each phoneme) can be input into the text encoder, and the phoneme encoding can be obtained through the text encoder and phoneme expansion.

[0073] Specifically, during the phoneme expansion process, the result output by the text encoder (i.e., the phoneme encoding result) can be copied according to the duration of the phoneme (i.e., the phoneme duration corresponding to each phoneme, such as the number of frames contained in the phoneme), and then the output result can be expanded from the phoneme level to the frame level to obtain the phoneme encoding.

[0074] In this embodiment, a plurality of phonemes and the pitch corresponding to each phoneme are input into a text encoder to obtain a phoneme encoding result, and then phoneme expansion is performed according to the phoneme encoding result and the phoneme duration corresponding to each phoneme to obtain the phoneme encoding, thereby providing data support for further energy-aware attention mechanism processing.

[0075] In one embodiment, based on the phoneme encoding and the energy encoding of the energy information to be input, a new energy encoding is obtained by processing through an energy-aware attention mechanism, which may include the following steps:

[0076] The phoneme encoding is used as a query for the energy-aware attention mechanism, and the energy encoding is used as the key and value for the energy-aware attention mechanism. After being processed by the energy-aware attention mechanism, a new energy encoding is obtained.

[0077] In the specific implementation, the phoneme encoding can be used as the Q (query) of the attention mechanism, that is, the query for the energy-aware attention mechanism, and the energy can be used as K and V (key and value), that is, the key and value for the energy-aware attention mechanism, and then the new energy encoding can be obtained through the energy-aware attention mechanism. Therefore, by inputting the phoneme encoding and energy into the energy-aware attention mechanism, the time alignment of the two can be automatically fine-tuned, which improves the robustness of the model.

[0078] In one example, the formula of the attention mechanism can be expressed as follows:

[0079]

[0080] Where Q is the query vector, K is the key vector, and V is the value vector; is the normalization factor, d k represents the dimension of the key vector, * represents matrix multiplication; the softmax function is used to normalize the similarity matrix to obtain the weight.

[0081] In this embodiment, by using the phoneme encoding as a query for the energy-aware attention mechanism, and the energy encoding as the key and value for the energy-aware attention mechanism, a new energy encoding is obtained after processing by the energy-aware attention mechanism, and the time alignment of the phoneme encoding and energy can be automatically fine-tuned, thereby improving the robustness of the model.

[0082] In one embodiment, the pre-trained singing synthesis model may further include a decoder, which generates a synthesized singing audio of a song to be synthesized according to the synthesized singing mel spectrum, and may include the following steps:

[0083] The phoneme coding and the new energy coding are fused to obtain a coding fusion result; the coding fusion result is processed by a decoder to obtain a synthetic singing mel spectrum.

[0084] In one example, the decoder in the pre-trained singing synthesis model can be constructed based on the corresponding module reference in the Fastspeech speech synthesis system, or other acoustic models can be used, which is not specifically limited in this embodiment. As shown in Figure 2, for the new energy code output by the energy-aware attention mechanism, it can be added to the phoneme code, that is, the phoneme code and the new energy code are fused to obtain the coding fusion result, and then the mel spectrum of the singing can be obtained through the decoder, that is, the synthesized singing mel spectrum.

[0085] In this embodiment, by fusing phoneme coding and new energy coding, a coding fusion result is obtained, and then the coding fusion result is processed by a decoder to obtain a synthesized singing mel-spectrogram, thereby realizing singing synthesis processing based on a highly robust and controllable energy-based singing synthesis system.

[0086] In one embodiment, the pre-trained singing voice synthesis model may further include an energy predictor or an energy extractor. In step S101, obtaining energy information to be input may include the following steps:

[0087] The energy information predicted based on the phoneme coding is used as the energy information to be input through an energy predictor; or, the energy information of the reference audio is extracted by an energy extractor as the energy information to be input.

[0088] In practical applications, for the singing synthesis process, the energy information can be obtained in different ways, such as from prediction and external provision. For example, as shown in Figure 2, when the energy prediction method is adopted, an energy predictor module can be added in the training model stage, and the L2 loss adjustment based on the corresponding predicted energy and real energy can be performed, and then the predicted energy information can be obtained in the singing synthesis application as the model input; for another example, when the external energy input method is adopted, a reference audio can be provided, and the energy information can be extracted from the reference audio as the model input through the energy extractor.

[0089] In an optional embodiment, due to the energy-aware attention mechanism, the length, singing content, and energy information of the reference audio do not need to be the same as those of the synthesized audio, and the generated singing energy can have a certain correlation with the energy of the input reference audio; the predicted energy or the external input energy can be modified to further control the energy of the synthesized audio.

[0090] In this embodiment, an energy predictor is used to obtain energy information based on phoneme coding prediction as energy information to be input, or an energy extractor is used to extract energy information of reference audio as energy information to be input. Energy information can be flexibly input, thereby improving the controllability of singing synthesis.

[0091] In one embodiment, the pre-trained singing synthesis model is trained by the following method:

[0092] Acquire training data; the training data includes multiple sample phonemes in the sample song and the sample pitch and sample phoneme duration corresponding to each sample phoneme, sample energy information of the sample song and sample singing mel spectrum; the sample singing mel spectrum is used as the sample label corresponding to the sample song; the multiple sample phonemes and the sample pitch and sample phoneme duration corresponding to each sample phoneme, and the sample energy information are input into the singing synthesis model to be trained to obtain the predicted singing mel spectrum; based on the loss of the predicted singing mel spectrum and the sample singing mel spectrum, the singing synthesis model to be trained is trained until the preset training end condition is met to obtain the pre-trained singing synthesis model.

[0093] In the specific implementation, the singing synthesis model training architecture shown in Figure 3 can be evolved from the FastSpeech speech synthesis system. By adding pitch and phoneme duration information, it can generate singing with a specified pitch, and the added energy-aware attention mechanism can jointly model energy and phoneme encoding.

[0094] In one example, the loss function of the model can be a Mel spectrum loss, such as the L2 loss between the predicted Mel spectrum and the real waveform extracted Mel spectrum; the energy-aware attention mechanism can be the attention mechanism in the transformer; the codecs used in the model are all stacked FFT (Feed-Forward Transformer) blocks, and the fastspeech-based codec can also be replaced with other modules, such as VITS, Diffusion, etc., which are not specifically limited in this embodiment.

[0095] In another example, the training data of the model may include singing audio data, the corresponding phoneme string, pitch, duration information, and energy information can be extracted from the singing audio data, that is, multiple sample phonemes and the sample pitch and sample phoneme duration corresponding to each sample phoneme, sample energy information of the sample song, and sample singing mel spectrum.

[0096] In the model training stage, as shown in FIG3 , phonemes, pitches, durations and energies can be used as inputs, wherein by using phonemes and pitches as inputs of the text encoder, the phoneme codes obtained after phoneme expansion can be output. For example, based on phoneme expansion, the results in the text encoder can be copied according to the duration of the phoneme (the number of frames contained in the phoneme), and the output can be expanded from the phoneme level to the frame level; then the phoneme codes and energies can be input into the attention mechanism, with the phoneme codes as the Q of the attention mechanism, and the energies as the K and V. The energy-aware attention mechanism automatically fine-tunes the time alignment of the two to improve robustness, and obtains a new energy code; and then the new energy code output after the attention mechanism can be added to the phoneme code and input into the decoder to obtain the Mel spectrum of the singing, that is, the predicted singing Mel spectrum, which can be further processed by a separately trained vocoder to obtain the predicted singing audio.

[0097] The technical solution of this embodiment adopts a highly energy-robust and controllable singing synthesis system to perform singing synthesis processing, and models energy based on the attention mechanism. It can automatically align the time relationship between energy and phonemes, and can avoid unstable model training caused by data problems such as inaccurate energy extraction and inaccurate phoneme duration labeling, thereby improving the robustness of the system.

[0098] In this embodiment, by acquiring training data, and then based on the loss of the predicted singing mel-spectrogram and the sample singing mel-spectrogram, the singing synthesis model to be trained is trained until the preset training end conditions are met, and a pre-trained singing synthesis model is obtained. This can avoid the instability of model training and effectively improve the robustness of the model.

[0099] In one embodiment, obtaining training data may include the following steps:

[0100] Acquire the singing audio data and sample music score information of the sample song; the sample music score information includes a plurality of sample phonemes and the sample pitch and sample phoneme duration corresponding to each sample phoneme; extract the sample energy information and the sample singing mel spectrum according to the singing audio data of the sample song.

[0101] In practical applications, for the training data processing in the training phase, training singing audio data and music scores can be obtained, that is, the singing audio data and sample music score information of the sample song, wherein the music score can contain phonemes, phoneme duration and pitch, that is, multiple sample phonemes and sample pitch and sample phoneme duration corresponding to each sample phoneme; Mel spectrum and energy can be extracted from the singing audio data, that is, sample singing Mel spectrum and sample energy information. After the data preparation is completed, each sample song can contain information such as phonemes, phoneme duration, pitch, Mel spectrum, energy, etc.

[0102] In this embodiment, by acquiring the singing audio data and sample music score information of the sample song, and then extracting the sample energy information and the sample singing mel spectrum according to the singing audio data of the sample song, data support is provided for model training.

[0103] In one embodiment, as shown in Figure 4, a flow chart of another singing voice synthesis method is provided. In this embodiment, the method comprises the following steps:

[0104] In step S401, training data is obtained, and a plurality of sample phonemes and sample pitches and sample phoneme durations corresponding to each sample phoneme, and sample energy information are input into a singing synthesis model to be trained to obtain a predicted singing mel-spectrogram. In step S402, based on the loss of the predicted singing mel-spectrogram and the sample singing mel-spectrogram, the singing synthesis model to be trained is trained until a preset training end condition is met to obtain a pre-trained singing synthesis model. In step S403, the score information of the song to be synthesized and the energy information to be input are obtained, and the score information and the energy information to be input are input into the pre-trained singing synthesis model. In step S404, the pre-trained singing synthesis model includes a text encoder, and a plurality of phonemes and the pitches corresponding to each phoneme are input into the text encoder to obtain a phoneme encoding result, and phoneme expansion is performed according to the phoneme encoding result and the phoneme duration corresponding to each phoneme to obtain a phoneme encoding. In step S405, the phoneme code is used as a query for the energy-aware attention mechanism, and the energy code is used as the key and value for the energy-aware attention mechanism. After being processed by the energy-aware attention mechanism, a new energy code is obtained. In step S406, the pre-trained singing synthesis model also includes a decoder, which fuses the phoneme code and the new energy code to obtain a coding fusion result, and the coding fusion result is processed by the decoder to obtain a synthesized singing mel spectrum. In step S407, based on the synthesized singing mel spectrum, a synthesized singing audio of the song to be synthesized is generated. It should be noted that the specific limitations of the above steps can be found in the specific limitations of a singing synthesis method above, which will not be repeated here.

[0105] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0106] Based on the same inventive concept, the embodiment of the present application also provides a singing voice synthesis device for implementing the singing voice synthesis method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more singing voice synthesis device embodiments provided below can refer to the limitations of the singing voice synthesis method above, and will not be repeated here.

[0107] In one embodiment, as shown in FIG5 , a singing voice synthesis device is provided, comprising:

[0108] The music score information and energy information acquisition module 501 is used to acquire the music score information of the song to be synthesized and the energy information to be input; the music score information includes a plurality of phonemes and the pitch and phoneme duration corresponding to each of the phonemes; the energy information to be input is predicted based on the music score information or extracted based on the reference audio of the song to be synthesized;

[0109] The singing synthesis model processing module 502 is used to input the music score information and the energy information to be input into the pre-trained singing synthesis model, and obtain the synthesized singing mel spectrum through the energy-aware attention mechanism in the pre-trained singing synthesis model; the energy-aware attention mechanism is used to combine the phoneme coding of the music score information and the energy coding of the energy information to be input to obtain a new energy coding;

[0110] The synthesized singing audio obtaining module 503 is used to generate the synthesized singing audio of the song to be synthesized according to the synthesized singing mel-spectrogram.

[0111] In one embodiment, the singing synthesis model processing module 502 includes:

[0112] A phoneme coding obtaining submodule, used for obtaining the phoneme coding according to the plurality of phonemes and the pitch and phoneme duration corresponding to each of the phonemes;

[0113] An energy-aware attention mechanism processing submodule, used for obtaining the new energy code through the energy-aware attention mechanism processing based on the phoneme code and the energy code of the energy information to be input;

[0114] The synthesized singing mel spectrum obtaining submodule is used to obtain the synthesized singing mel spectrum according to the encoding fusion result of the phoneme encoding and the new energy encoding.

[0115] In one embodiment, the pre-trained singing synthesis model includes a text encoder, and the phoneme encoding submodule includes:

[0116] A text encoder processing unit, used for inputting the plurality of phonemes and the pitch corresponding to each of the phonemes into the text encoder to obtain a phoneme encoding result;

[0117] The phoneme expansion unit is used to perform phoneme expansion according to the phoneme encoding result and the phoneme duration corresponding to each of the phonemes to obtain the phoneme encoding.

[0118] In one embodiment, the energy-aware attention mechanism processing submodule includes:

[0119] A new energy coding obtaining unit is used to use the phoneme coding as a query for the energy-aware attention mechanism, and use the energy coding as a key and value for the energy-aware attention mechanism, and obtain the new energy coding after processing by the energy-aware attention mechanism.

[0120] In one embodiment, the pre-trained singing synthesis model further includes an energy predictor or an energy extractor, and the synthesized singing mel spectrum obtaining submodule includes:

[0121] A coding fusion unit, used for fusing the phoneme coding and the new energy coding to obtain the coding fusion result;

[0122] The decoder processing unit is used to process the encoding fusion result through the decoder to obtain the synthesized singing mel spectrum.

[0123] In one embodiment, the music score information and energy information acquisition module 501 includes:

[0124] The module for acquiring energy information to be input is used to use the energy predictor to obtain energy information based on the phoneme encoding prediction as the energy information to be input; or to use the energy extractor to extract energy information of the reference audio as the energy information to be input.

[0125] In one embodiment, the pre-trained singing synthesis model is trained by the following method:

[0126] A training data acquisition module, used to acquire training data; the training data includes a plurality of sample phonemes in a sample song, a sample pitch and a sample phoneme duration corresponding to each of the sample phonemes, sample energy information of the sample song, and a sample singing mel spectrum; the sample singing mel spectrum is used as a sample label corresponding to the sample song;

[0127] A module for obtaining a predicted singing mel spectrum, which is used to input the plurality of sample phonemes, the sample pitch and the sample phoneme duration corresponding to each of the sample phonemes, and the sample energy information into a singing synthesis model to be trained, to obtain a predicted singing mel spectrum;

[0128] The model training module is used to train the singing synthesis model to be trained based on the loss of the predicted singing mel-spectrogram and the sample singing mel-spectrogram until the preset training end condition is met to obtain the pre-trained singing synthesis model.

[0129] The sample acquisition submodule is used to acquire the singing audio data and sample music score information of the sample song; the sample music score information includes the multiple sample phonemes and the sample pitch and sample phoneme duration corresponding to each of the sample phonemes;

[0130] The extraction submodule is used to extract the sample energy information and the sample singing mel spectrum according to the singing audio data of the sample song.

[0131] Each module in the above-mentioned singing voice synthesis device can be implemented in whole or in part by software, hardware and a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module above.

[0132] In an exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be shown in FIG6. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit, and an input device. The processor, the memory, and the input / output interface are connected via a system bus, and the communication interface, the display unit, and the input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a singing synthesis method is implemented.

[0133] Those skilled in the art will understand that the structure shown in FIG. 6 is merely a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0134] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0135] Obtaining music score information of a song to be synthesized and energy information to be input; the music score information includes a plurality of phonemes and the pitch and duration of each phoneme; the energy information to be input is predicted based on the music score information or extracted based on the reference audio of the song to be synthesized;

[0136] The music score information and the energy information to be input are input into a pre-trained singing synthesis model, and the synthesized singing mel spectrum is obtained by processing the energy-aware attention mechanism in the pre-trained singing synthesis model; the energy-aware attention mechanism is used to combine the phoneme encoding of the music score information and the energy encoding of the energy information to be input to obtain a new energy encoding;

[0137] According to the synthesized singing mel-spectrogram, the synthesized singing audio of the song to be synthesized is generated.

[0138] In one embodiment, when the processor executes the computer program, the steps in the other embodiments described above are also implemented.

[0139] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0140] Obtaining music score information of a song to be synthesized and energy information to be input; the music score information includes a plurality of phonemes and the pitch and duration of each phoneme; the energy information to be input is predicted based on the music score information or extracted based on the reference audio of the song to be synthesized;

[0141] The music score information and the energy information to be input are input into a pre-trained singing synthesis model, and the synthesized singing mel spectrum is obtained by processing the energy-aware attention mechanism in the pre-trained singing synthesis model; the energy-aware attention mechanism is used to combine the phoneme encoding of the music score information and the energy encoding of the energy information to be input to obtain a new energy encoding;

[0142] According to the synthesized singing mel-spectrogram, the synthesized singing audio of the song to be synthesized is generated.

[0143] In one embodiment, when the computer program is executed by a processor, the steps in the other embodiments described above are also implemented.

[0144] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:

[0145] Obtaining music score information of a song to be synthesized and energy information to be input; the music score information includes a plurality of phonemes and the pitch and duration of each phoneme; the energy information to be input is predicted based on the music score information or extracted based on the reference audio of the song to be synthesized;

[0146] The music score information and the energy information to be input are input into a pre-trained singing synthesis model, and the synthesized singing mel spectrum is obtained by processing the energy-aware attention mechanism in the pre-trained singing synthesis model; the energy-aware attention mechanism is used to combine the phoneme encoding of the music score information and the energy encoding of the energy information to be input to obtain a new energy encoding;

[0147] According to the synthesized singing mel-spectrogram, the synthesized singing audio of the song to be synthesized is generated.

[0148] In one embodiment, when the computer program is executed by a processor, the steps in the other embodiments described above are also implemented.

[0149] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0150] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0151] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0152] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A singing voice synthesis method, It is characterized in that The method comprises: Obtaining music score information of a song to be synthesized and energy information to be input; the music score information includes a plurality of phonemes and the pitch and duration of each phoneme; the energy information to be input is predicted based on the music score information or extracted based on the reference audio of the song to be synthesized; The music score information and the energy information to be input are input into a pre-trained singing synthesis model, and the synthesized singing mel spectrum is obtained by processing the energy-aware attention mechanism in the pre-trained singing synthesis model; the energy-aware attention mechanism is used to combine the phoneme encoding of the music score information and the energy encoding of the energy information to be input to obtain a new energy encoding; According to the synthesized singing mel-spectrogram, the synthesized singing audio of the song to be synthesized is generated.

2. The method according to claim 1, It is characterized in that The step of inputting the music score information and the energy information to be input into a pre-trained singing synthesis model, and processing the pre-trained singing synthesis model through an energy-aware attention mechanism to obtain a synthesized singing mel spectrum comprises: Obtaining the phoneme code according to the plurality of phonemes and the pitch and phoneme duration corresponding to each of the phonemes; Based on the phoneme code and the energy code of the energy information to be input, the new energy code is obtained by processing the energy-aware attention mechanism; The synthesized singing mel spectrum is obtained according to the encoding fusion result of the phoneme encoding and the new energy encoding.

3. The method according to claim 2, It is characterized in that The pre-trained singing synthesis model includes a text encoder, and the phoneme encoding is obtained according to the plurality of phonemes and the pitch and phoneme duration corresponding to each of the phonemes, including: Inputting the plurality of phonemes and the pitch corresponding to each of the phonemes into the text encoder to obtain a phoneme encoding result; Phoneme expansion is performed according to the phoneme encoding result and the phoneme duration corresponding to each of the phonemes to obtain the phoneme encoding.

4. The method according to claim 2, It is characterized in that The energy encoding based on the phoneme encoding and the energy information to be input is processed by the energy-aware attention mechanism to obtain the new energy encoding, including: The phoneme encoding is used as a query for the energy-aware attention mechanism, and the energy encoding is used as a key and value for the energy-aware attention mechanism. The new energy encoding is obtained by processing the energy-aware attention mechanism.

5. The method according to claim 2, It is characterized in that The pre-trained singing synthesis model also includes a decoder, and the encoding fusion result according to the phoneme encoding and the new energy encoding is used to obtain the synthesized singing mel spectrum, including: Fusing the phoneme code and the new energy code to obtain the code fusion result; The encoding fusion result is processed by the decoder to obtain the synthesized singing mel spectrum.

6. The method according to claim 1, It is characterized in that The pre-trained singing voice synthesis model further includes an energy predictor or an energy extractor, and the step of obtaining energy information to be input includes: By means of the energy predictor, energy information predicted based on the phoneme coding is used as the energy information to be input; Alternatively, the energy extractor is used to extract energy information of the reference audio as the energy information to be input.

7. The method according to any one of claims 1 to 6, It is characterized in that The pre-trained singing synthesis model is trained by the following method: Acquire training data; the training data includes a plurality of sample phonemes in a sample song, a sample pitch and a sample phoneme duration corresponding to each of the sample phonemes, sample energy information of the sample song, and a sample singing Mel spectrum; the sample singing Mel spectrum is used as a sample label corresponding to the sample song; Input the plurality of sample phonemes, the sample pitch and the sample phoneme duration corresponding to each of the sample phonemes, and the sample energy information into the singing synthesis model to be trained to obtain a predicted singing mel spectrum; Based on the loss between the predicted singing mel-spectrogram and the sample singing mel-spectrogram, the singing synthesis model to be trained is trained until a preset training end condition is met, thereby obtaining the pre-trained singing synthesis model.

8. The method according to claim 7, It is characterized in that The obtaining of training data comprises: Acquire singing audio data and sample music score information of the sample song; the sample music score information includes the multiple sample phonemes and the sample pitch and sample phoneme duration corresponding to each of the sample phonemes; The sample energy information and the sample singing mel spectrum are extracted based on the singing audio data of the sample song.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program. It is characterized in that When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images