Song synthesis method, computer device, and computer-readable storage medium

Abstract

The application relates to a singing synthesis method, a computer device and a computer readable storage medium. The method comprises the following steps: obtaining score information of a to-be-synthesized song and to-be-input energy information; the score information comprises a plurality of phonemes and corresponding pitch and phoneme duration of each phoneme; the to-be-input energy information is obtained based on prediction of the score information or extraction based on reference audio of the to-be-synthesized song; the score information and the to-be-input energy information are input into a pre-trained singing synthesis model, and a synthesized singing mel-spectrogram is obtained through energy-aware attention mechanism processing in the pre-trained singing synthesis model; the energy-aware attention mechanism is used to combine phoneme coding of the score information and energy coding of the to-be-input energy information to obtain new energy coding; and a synthesized singing audio of the to-be-synthesized song is generated according to the synthesized singing mel-spectrogram. The method can effectively improve the stability of the model and the controllability of singing synthesis, and is helpful to improve the singing synthesis effect.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of song synthesis, in particular to a song synthesis method, a computer device and a computer readable storage medium. BACKGROUND

[0002] With the development of Internet technology, song synthesis technology appears, which can synthesize human-like song by inputting score. Since there are various singing skills of song, the singing of word biting is an important skill, and the energy of audio is closely related to the biting of words, so energy modeling is crucial in the song synthesis system.

[0003] In the traditional technology, due to the inaccurate extraction of energy due to external noise and other factors, and the possible deviation of phoneme duration annotation, the model robustness is low, and the song synthesis effect is poor. SUMMARY

[0004] Therefore, it is necessary to provide a song synthesis method, a computer device and a computer readable storage medium capable of improving the robustness of the song synthesis system.

[0005] In a first aspect, the present application provides a song synthesis method. The method comprises:

[0006] obtaining score information of a to-be-synthesized song and to-be-input energy information; the score information comprises a plurality of phonemes and a pitch and a phoneme duration corresponding to each phoneme; the to-be-input energy information is obtained based on the score information or extracted based on reference audio of the to-be-synthesized song;

[0007] inputting the score information and the to-be-input energy information into a pre-trained song synthesis model, and obtaining a synthesized song mel-spectrogram through energy-aware attention mechanism processing in the pre-trained song synthesis model; the energy-aware attention mechanism is used to combine phoneme encoding of the score information and energy encoding of the to-be-input energy information to obtain new energy encoding;

[0008] generating a synthesized song audio of the to-be-synthesized song according to the synthesized song mel-spectrogram.

[0009] In one embodiment, the inputting the score information and the to-be-input energy information into a pre-trained song synthesis model, and obtaining a synthesized song mel-spectrogram through energy-aware attention mechanism processing in the pre-trained song synthesis model comprises:

[0010] obtaining the phoneme encoding according to the plurality of phonemes and the pitch and the phoneme duration corresponding to each phoneme;

[0011] The energy encoding is processed through the energy-aware attention mechanism based on the phoneme encoding and the to-be-input energy information to obtain the new energy encoding.

[0012] The synthesized singing mel-spectrogram is obtained according to the encoding fusion result of the phoneme encoding and the new energy encoding.

[0013] In one of the embodiments, the pre-trained singing synthesis model comprises a text encoder, and the phoneme encoding is obtained according to the plurality of phonemes and the corresponding pitch and phoneme duration of each phoneme, comprising:

[0014] The plurality of phonemes and the corresponding pitch of each phoneme are input into the text encoder to obtain a phoneme encoding result.

[0015] The phoneme encoding is obtained according to the phoneme encoding result and the corresponding phoneme duration of each phoneme.

[0016] In one of the embodiments, the energy encoding is processed through the energy-aware attention mechanism based on the phoneme encoding and the to-be-input energy information to obtain the new energy encoding, comprising:

[0017] The phoneme encoding is taken as a query for the energy-aware attention mechanism, the energy encoding is taken as a key and a value for the energy-aware attention mechanism, and the energy-aware attention mechanism is processed to obtain the new energy encoding.

[0018] In one of the embodiments, the pre-trained singing synthesis model further comprises a decoder, and the synthesized singing mel-spectrogram is obtained according to the encoding fusion result of the phoneme encoding and the new energy encoding, comprising:

[0019] The phoneme encoding and the new energy encoding are fused to obtain the encoding fusion result.

[0020] The encoding fusion result is processed by the decoder to obtain the synthesized singing mel-spectrogram.

[0021] In one of the embodiments, the pre-trained singing synthesis model further comprises an energy predictor or an energy extractor, and the to-be-input energy information is obtained, comprising:

[0022] The energy information predicted based on the phoneme encoding is taken as the to-be-input energy information by the energy predictor.

[0023] Or, the energy information of the reference audio is extracted by the energy extractor and taken as the to-be-input energy information.

[0024] In one of the embodiments, the pre-trained singing synthesis model is trained by the following method:

[0025] obtaining training data; the training data comprises a plurality of sample phonemes in a sample song, sample pitch and sample phoneme duration corresponding to each of the sample phonemes, sample energy information and a sample singing mel-spectrogram of the sample song; the sample singing mel-spectrogram is used as a sample label corresponding to the sample song;

[0026] inputting the plurality of sample phonemes, sample pitch and sample phoneme duration corresponding to each of the sample phonemes, and the sample energy information into a singing synthesis model to be trained to obtain a predicted singing mel-spectrogram;

[0027] training the singing synthesis model to be trained based on a loss of the predicted singing mel-spectrogram and the sample singing mel-spectrogram until a preset training end condition is met, to obtain the pre-trained singing synthesis model.

[0028] In one of the embodiments, the obtaining training data comprises:

[0029] obtaining song audio data and sample score information of the sample song; the sample score information comprises the plurality of sample phonemes and sample pitch and sample phoneme duration corresponding to each of the sample phonemes;

[0030] extracting the sample energy information and the sample singing mel-spectrogram from the song audio data of the sample song.

[0031] In a second aspect, the present application further provides a computer device. The computer device comprises a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0032] obtaining score information of a song to be synthesized and input energy information; the score information comprises a plurality of phonemes and pitch and phoneme duration corresponding to each of the phonemes; the input energy information is obtained based on the score information or extracted based on reference audio of the song to be synthesized;

[0033] inputting the score information and the input energy information into a pre-trained singing synthesis model, and processing by an energy-aware attention mechanism in the pre-trained singing synthesis model to obtain a synthesized singing mel-spectrogram; the energy-aware attention mechanism is used to combine phoneme encoding of the score information and energy encoding of the input energy information to obtain new energy encoding;

[0034] generating a synthesized song audio of the song to be synthesized according to the synthesized singing mel-spectrogram.

[0035] In a third aspect, the present application also provides a computer readable storage medium. The computer readable storage medium has a computer program stored thereon, and the computer program, when executed by a processor, implements the following steps:

[0036] obtaining score information of a to-be-synthesized song and to-be-input energy information; the score information comprises a plurality of phonemes and a pitch and a phoneme duration corresponding to each phoneme; the to-be-input energy information is obtained based on the score information or extracted based on reference audio of the to-be-synthesized song;

[0037] inputting the score information and the to-be-input energy information into a pre-trained singing synthesis model, and obtaining a synthesized singing mel-spectrogram through an energy-aware attention mechanism in the pre-trained singing synthesis model; the energy-aware attention mechanism is used to combine phoneme encoding of the score information and energy encoding of the to-be-input energy information to obtain new energy encoding;

[0038] generating a synthesized singing audio of the to-be-synthesized song according to the synthesized singing mel-spectrogram.

[0039] In a fourth aspect, the present application also provides a computer program product. The computer program product comprises a computer program, and the computer program, when executed by a processor, implements the following steps:

[0040] obtaining score information of a to-be-synthesized song and to-be-input energy information; the score information comprises a plurality of phonemes and a pitch and a phoneme duration corresponding to each phoneme; the to-be-input energy information is obtained based on the score information or extracted based on reference audio of the to-be-synthesized song;

[0041] inputting the score information and the to-be-input energy information into a pre-trained singing synthesis model, and obtaining a synthesized singing mel-spectrogram through an energy-aware attention mechanism in the pre-trained singing synthesis model; the energy-aware attention mechanism is used to combine phoneme encoding of the score information and energy encoding of the to-be-input energy information to obtain new energy encoding;

[0042] generating a synthesized singing audio of the to-be-synthesized song according to the synthesized singing mel-spectrogram.

[0043] The aforementioned method, computer device, and computer-readable storage medium for synthesizing vocal data acquire the score information of the song to be synthesized, as well as the energy information to be input. The score information includes multiple phonemes and their corresponding pitches and durations. The energy information to be input is predicted based on the score information or extracted from a reference audio of the song to be synthesized. The score information and the energy information to be input are then input into a pre-trained vocal synthesis model. The model processes the data through an energy-aware attention mechanism to obtain a synthesized vocal spectrum. This energy-aware attention mechanism combines the phoneme encoding of the score information with the energy encoding of the energy information to obtain a new energy encoding. Based on the synthesized vocal spectrum, the synthesized vocal audio of the song to be synthesized is generated. This optimizes the vocal synthesis process. The energy-aware attention mechanism effectively improves the model's stability by processing the phoneme encoding of the score information and the energy encoding of the energy information to be input, allowing for flexible input of energy information and enhancing the controllability of vocal synthesis, thus contributing to improved vocal synthesis results. Attached Figure Description

[0044] Figure 1 This is a flowchart illustrating a singing voice synthesis method in one embodiment;

[0045] Figure 2 This is a schematic diagram of a singing voice synthesis processing application architecture in one embodiment;

[0046] Figure 3 This is a schematic diagram of a vocal synthesis processing training architecture in one embodiment;

[0047] Figure 4 This is a flowchart illustrating the singing voice synthesis method in another embodiment;

[0048] Figure 5 This is a structural block diagram of a singing voice synthesis method apparatus in one embodiment;

[0049] Figure 6 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0051] In one embodiment, such as Figure 1As shown, a singing voice synthesis method is provided. In this embodiment, the method is applied to a terminal. It should be understood that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction of the terminal and the server. The server can be implemented by an independent server or a server cluster composed of multiple servers. The terminal can be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers, and portable wearable devices.

[0052] The method in this embodiment includes the following steps S101 to S103.

[0053] In step S101, score information of a song to be synthesized and input energy information are obtained.

[0054] The score information can include multiple phonemes and the pitch and phoneme duration corresponding to each phoneme.

[0055] As an example, the input energy information can be predicted based on the score information or extracted based on a reference audio of the song to be synthesized.

[0056] In actual applications, the singing voice synthesis processing can be performed based on an energy-aware singing voice synthesis system. For example, based on the singing voice synthesis function in an APP or a client, the SVS (Singing-voice-synthesis) technology can be used. The score information of a song to be synthesized can be obtained, which can include but is not limited to phonemes, pitch, duration, i.e., multiple phonemes and the pitch and phoneme duration corresponding to each phoneme. In combination with the input energy information, the singing voice of the specified score content can be output.

[0057] In an example, for a singing voice audio, the energy of the audio is related to the size of the audio sound. The greater the energy, the larger the sound. The energy of the audio can be extracted from the audio waveform file as a feature of the audio, i.e., energy information.

[0058] For example, when a user needs to synthesize a singing voice, the singing voice synthesis processing can be performed through the singing voice synthesis function in an APP or a client. The score information of a song to be synthesized can be input, and the input of a reference audio can be provided. The reference audio can be used to extract energy information according to the reference audio, so that the energy of the output singing voice and the energy of the reference audio have a certain similarity. The input energy information can also be obtained by predicting the energy, i.e., predicted based on the score information or extracted based on the reference audio of the song to be synthesized. Thus, the input of the energy is flexible, and can also be manually modified, which can improve the controllability of the singing voice synthesis.

[0059] In step S102, the musical score information and the energy information to be input are input into the pre-trained singing synthesis model. The energy perception attention mechanism in the pre-trained singing synthesis model is used to process the synthesized singing Mel spectrum.

[0060] Among them, the energy-sensing attention mechanism can be used to combine the phoneme encoding of the musical score information and the energy encoding of the energy information to be input to obtain a new energy encoding, which is obtained after aligning the temporal relationship between energy and phonemes.

[0061] In the process of singing synthesis, such as Figure 2 As shown, phoneme strings, pitches, phoneme durations (i.e., multiple phonemes and their corresponding pitches and durations), and energy can be used as inputs. That is, the musical score information and the energy information to be input are fed into a pre-trained singing synthesis model. Based on the multiple phonemes and their corresponding pitches and durations, phoneme codes are obtained. Then, based on the phoneme codes and the energy codes of the input energy information, a new energy code is obtained through an energy-aware attention mechanism. Finally, based on the encoding fusion result of the phoneme codes and the new energy codes, the corresponding Mel spectrum of the singing (i.e., the synthesized singing Mel spectrum) is synthesized. Thus, by combining energy and phonemes through an attention mechanism, the robustness and controllability of the system for energy modeling are improved.

[0062] In step S103, the synthesized vocal audio of the song to be synthesized is generated based on the synthesized vocal spectrum.

[0063] After obtaining the synthesized vocal spectrum, the synthesized vocal spectrum can be processed by a separately trained vocoder to obtain the corresponding vocal audio, that is, the synthesized vocal audio of the song to be synthesized.

[0064] Compared to traditional methods, such as energy-based speech synthesis, which directly adds energy to phoneme encoding, the model's robustness is low because energy extraction may be inaccurate due to external noise and phoneme duration annotation may also be biased. Similarly, traditional vocal synthesis systems that do not model energy suffer from low energy control due to the lack of energy input, resulting in smooth and random variations in synthesized vocal energy. The technical solution in this embodiment employs a highly robust and controllable vocal synthesis system. Based on an energy-aware attention mechanism, it improves system robustness and provides an energy input point, allowing the use of energy not only automatically predicted by the model but also from externally input reference audio. Furthermore, it eliminates restrictions on the duration and content of the reference audio, enhancing control over the model's energy.

[0065] In the above-mentioned singing synthesis method, the musical score information and the energy information to be input for the song to be synthesized are obtained. Then, the musical score information and the energy information to be input are input into a pre-trained singing synthesis model. The energy-aware attention mechanism in the pre-trained singing synthesis model is used to process the synthesized singing Mel spectrum. Then, based on the synthesized singing Mel spectrum, the synthesized singing audio of the song to be synthesized is generated. This optimizes the singing synthesis process. The phoneme encoding of the musical score information and the energy encoding of the energy information to be input are processed based on the energy-aware attention mechanism, which can effectively improve the stability of the model and can flexibly input energy information, thereby improving the controllability of singing synthesis and helping to improve the singing synthesis effect.

[0066] In one embodiment, in step S102, the musical score information and the energy information to be input are input into a pre-trained singing synthesis model. The synthesized singing Mel spectrum is obtained through the energy-aware attention mechanism in the pre-trained singing synthesis model. This may include the following steps:

[0067] Based on multiple phonemes and their corresponding pitch and duration, a phoneme code is obtained; based on the phoneme code and the energy code of the energy information to be input, a new energy code is obtained through energy perception attention mechanism; based on the encoding fusion result of the phoneme code and the new energy code, a synthesized singing melodic spectrum is obtained.

[0068] In specific implementations, such as Figure 2 The pre-trained vocal synthesis model architecture shown in the energy-aware vocal synthesis system can include a text encoder, an energy-aware attention mechanism, and a decoder. During vocal synthesis, the musical score information (including multiple phonemes and their corresponding pitch and duration, such as...) can be used. Figure 2 The phoneme string (pitch, phoneme duration) is input into the pre-trained singing synthesis model. After passing through the text encoder and phoneme expansion, the phoneme code can be obtained. Then, the phoneme code can be combined with the predicted energy or the energy extracted from the reference audio (i.e. the energy information to be input) and input into the energy-aware attention mechanism to obtain a new energy code. Then, the new energy code can be added to the phoneme code and processed by the decoder to synthesize the corresponding singing Mel spectrum (i.e., the synthesized singing Mel spectrum).

[0069] In this embodiment, phoneme encoding is obtained based on multiple phonemes and their corresponding pitch and duration. Then, based on the phoneme encoding and the energy encoding of the energy information to be input, a new energy encoding is obtained through energy-aware attention mechanism. Finally, based on the encoding fusion result of the phoneme encoding and the new energy encoding, the synthesized singing melodic spectrum is obtained, which can effectively improve the stability of the model and help improve the singing synthesis effect.

[0070] In one embodiment, the pre-trained song synthesis model can include a text encoder, obtain phoneme encodings according to a plurality of phonemes and corresponding pitch and phoneme duration of each phoneme, and can include the following steps:

[0071] The plurality of phonemes and the corresponding pitch of each phoneme are input into the text encoder to obtain phoneme encoding results; phoneme expansion is performed according to the phoneme encoding results and the corresponding phoneme duration of each phoneme to obtain phoneme encodings.

[0072] In an example, the text encoder in the pre-trained song synthesis model can be constructed based on the corresponding module in the Fastspeech speech synthesis system, or other acoustic models can be used, and the specific implementation is not limited in the embodiment. As shown in Figure 2 The phoneme string and the pitch (i.e., the plurality of phonemes and the corresponding pitch of each phoneme) are input into the text encoder, and the phoneme encoding is obtained through the text encoder and the phoneme expansion.

[0073] Specifically, in the phoneme expansion process, the result output by the text encoder (i.e., the phoneme encoding result) can be copied according to the duration of the phoneme (i.e., the corresponding phoneme duration of each phoneme, such as the number of frames contained in the phoneme), and the output result can be expanded from the phoneme level to the frame level to obtain the phoneme encoding.

[0074] In the embodiment, the plurality of phonemes and the corresponding pitch of each phoneme are input into the text encoder to obtain the phoneme encoding result, and then the phoneme expansion is performed according to the phoneme encoding result and the corresponding phoneme duration of each phoneme to obtain the phoneme encoding, which provides data support for further energy-aware attention mechanism processing.

[0075] In one embodiment, based on the phoneme encoding and the energy encoding of the to-be-input energy information, a new energy encoding is obtained through energy-aware attention mechanism processing, which can include the following steps:

[0076] The phoneme encoding is used as a query for the energy-aware attention mechanism, and the energy encoding is used as a key and a value for the energy-aware attention mechanism, and the energy-aware attention mechanism is processed to obtain a new energy encoding.

[0077] In a specific implementation, the phoneme encoding can be used as the Q (query) of the attention mechanism, i.e., the query for the energy-aware attention mechanism, and the energy can be used as the K and V (key and value), i.e., the key and value for the energy-aware attention mechanism, and then the energy-aware attention mechanism can be used to obtain a new energy encoding. Thus, by inputting the phoneme encoding and the energy into the energy-aware attention mechanism, the time alignment of the two can be automatically fine-tuned, and the model robustness is improved.

[0078] In an example, the formula of the attention mechanism can be expressed in the following way:

[0079]

[0080] wherein Q is the query vector, K is the key vector, and V is the value vector; is a normalization factor, d k represents the dimension of the key vector, * represents matrix multiplication; the softmax function is used to normalize the similarity matrix to obtain the weight.

[0081] In the embodiment, by taking the phoneme code as the query for the energy-aware attention mechanism and the energy code as the key and value for the energy-aware attention mechanism, the new energy code is obtained by processing through the energy-aware attention mechanism, which can automatically fine-tune the time alignment of the phoneme code and the energy, thereby improving the robustness of the model.

[0082] In an embodiment, the pre-trained song synthesis model can further include a decoder configured to generate a synthesized song audio of the to-be-synthesized song according to the synthesized song mel-spectrogram, and can include the following steps:

[0083] The phoneme code and the new energy code are fused to obtain a code fusion result; and the code fusion result is processed by the decoder to obtain a synthesized song mel-spectrogram.

[0084] In an example, the decoder in the pre-trained song synthesis model can be constructed based on the corresponding module in the Fastspeech speech synthesis system, or other acoustic models can be used, which are not specifically limited in the embodiment. As shown in Figure 2 The new energy code output by the energy-aware attention mechanism can be added to the phoneme code, i.e., the phoneme code and the new energy code are fused to obtain a code fusion result, and then the mel-spectrogram of the song, i.e., the synthesized song mel-spectrogram, can be obtained through the decoder.

[0085] In the embodiment, the phoneme code and the new energy code are fused to obtain a code fusion result, and then the code fusion result is processed by the decoder to obtain a synthesized song mel-spectrogram, thereby realizing song synthesis processing based on an energy-aware and controllable song synthesis system.

[0086] In an embodiment, the pre-trained song synthesis model can further include an energy predictor or an energy extractor, and in step S101, the to-be-input energy information can be obtained, which can include the following steps:

[0087] The energy information predicted based on the phoneme coding is input as the to-be-input energy information by the energy predictor, or the energy information of the reference audio is extracted as the to-be-input energy information by the energy extractor.

[0088] In actual applications, for a singing synthesis process, the energy information can be obtained in different ways, such as from prediction or external provision. For example, as shown in FIG. 6, when the predicted energy is used, an energy predictor module can be added in the training model stage, and the predicted energy information can be obtained as the model input based on the L2 loss adjustment of the corresponding predicted energy and the real energy in the singing synthesis application; or, as shown in FIG. 7, when the external input energy is used, a reference audio can be provided, and the energy information of the reference audio can be extracted as the model input by the energy extractor. Figure 2

[0089] In an optional embodiment, due to the energy-aware attention mechanism, the length of the reference audio, the singing content, and the energy information do not need to be the same as the synthesized audio, and the generated singing energy can be related to the energy of the input reference audio; the predicted energy or the externally input energy can be further controlled to control the energy of the synthesized audio.

[0090] In the embodiment, the energy information predicted based on the phoneme coding is input as the to-be-input energy information by the energy predictor, or the energy information of the reference audio is extracted as the to-be-input energy information by the energy extractor, so that the energy information can be flexibly input, and the controllability of the singing synthesis is improved.

[0091] In an embodiment, the pre-trained singing synthesis model is obtained by the following method:

[0092] obtaining training data; the training data includes a plurality of sample phonemes in a sample song, sample pitch and sample phoneme duration corresponding to each sample phoneme, sample energy information and a sample singing mel-spectrogram of the sample song; the sample singing mel-spectrogram is used as a sample label corresponding to the sample song; the plurality of sample phonemes, the sample pitch and the sample phoneme duration corresponding to each sample phoneme, and the sample energy information are input into a to-be-trained singing synthesis model to obtain a predicted singing mel-spectrogram; the to-be-trained singing synthesis model is trained based on a loss of the predicted singing mel-spectrogram and the sample singing mel-spectrogram until a preset training end condition is met, and a pre-trained singing synthesis model is obtained.

[0093] In a specific implementation, as shown in FIG. 8, the pre-trained singing synthesis model is obtained by the following method: Figure 3 ​The illustrated song synthesis model training architecture, which can be evolved from a FastSpeech speech synthesis system, is capable of generating songs with specified pitches by adding pitch and phoneme duration information, and is capable of jointly modeling energy and phoneme encodings by adding an energy-aware attention mechanism.

[0094] In an example, the loss function of the model can be a mel-spectrogram loss, such as an L2 loss between predicted mel-spectrograms and ground-truth waveform-extracted mel-spectrograms; the energy-aware attention mechanism can be an attention mechanism in a transformer; both the encoder and the decoder used by the model are stacked FFT (Feed-Forward Transformer) blocks, and other modules such as VITS, Diffusion, etc. can be used to replace the fastspeech-based encoder and decoder, which are not specifically limited in this embodiment.

[0095] In yet another example, the training data of the model can include song audio data, corresponding phoneme strings, pitch, duration information, and energy information, which can be extracted from the song audio data, i.e., multiple sample phonemes and corresponding sample pitches and sample phoneme durations, sample energy information and sample song mel-spectrograms of the sample song.

[0096] In the model training phase, as shown in Figure 3 The phonemes, pitches, durations, and energies can be used as inputs, wherein the phonemes and pitches can be used as inputs of a text encoder to output phoneme encodings obtained by phoneme expansion, e.g., based on phoneme expansion, the results in the text encoder can be copied according to the duration of the phonemes (the number of frames contained by the phonemes) to expand the output from the phoneme level to the frame level; then the phoneme encodings and the energies can be input into an attention mechanism, the phoneme encodings can be used as Q and the energies can be used as K and V, and the energy-aware attention mechanism can automatically fine-tune the time alignment of the two to improve robustness to obtain new energy encodings; further, the new energy encodings output after the attention mechanism can be added to the phoneme encodings and input into a decoder to obtain a mel-spectrogram of the song, i.e., a predicted song mel-spectrogram, which can be further processed by a separately trained vocoder to obtain a predicted song audio.

[0097] The technical solution of the present embodiment uses a song synthesis system with high robustness and controllable energy to perform song synthesis processing, and models the energy based on an attention mechanism to automatically align the time relationship between the energy and the phonemes, which can avoid unstable model training caused by inaccurate energy extraction and inaccurate phoneme duration annotation, and improves the robustness of the system.

[0098] In the embodiment, the song synthesis model to be trained is trained by obtaining training data and then based on the loss of the predicted song mel-spectrogram and the sample song mel-spectrogram until a preset training end condition is met, to obtain the pre-trained song synthesis model, so that the model training instability can be avoided, and the robustness of the model is effectively improved.

[0099] In one embodiment, obtaining the training data can include the following steps:

[0100] The song audio data and sample score information of the sample song are obtained, the sample score information includes a plurality of sample phonemes and sample pitch and sample phoneme duration corresponding to each sample phoneme, and the sample energy information and sample song mel-spectrogram are extracted according to the song audio data of the sample song.

[0101] In actual application, for the training data processing in the training stage, the training song audio data and score can be obtained, that is, the song audio data and sample score information of the sample song, wherein the score can include phonemes, phoneme duration and pitch, that is, a plurality of sample phonemes and sample pitch and sample phoneme duration corresponding to each sample phoneme; the mel-spectrogram and energy can be extracted from the song audio data, that is, the sample song mel-spectrogram and sample energy information. After the data preparation is completed, each sample song can include phoneme, phoneme duration, pitch, mel-spectrogram, energy and other information.

[0102] In the embodiment, the song audio data and sample score information of the sample song are obtained, and then the sample energy information and sample song mel-spectrogram are extracted according to the song audio data of the sample song, to provide data support for model training.

[0103] In one embodiment, as shown in Figure 4 Another flowchart of a song synthesis method is provided. In the embodiment, the method includes the following steps:

[0104] In step S401, training data is obtained, a plurality of sample phonemes, and sample pitch and sample phoneme duration and sample energy information corresponding to each sample phoneme are input to the song synthesis model to be trained to obtain a predicted song mel-spectrum. In step S402, the song synthesis model to be trained is trained based on the loss of the predicted song mel-spectrum and the sample song mel-spectrum until a preset training end condition is met, and a pre-trained song synthesis model is obtained. In step S403, the score information of the song to be synthesized and the input energy information are obtained, and the score information and the input energy information are input to the pre-trained song synthesis model. In step S404, the pre-trained song synthesis model includes a text encoder, a plurality of phonemes and a pitch corresponding to each phoneme are input to the text encoder to obtain a phoneme encoding result, and the phoneme is expanded according to the phoneme encoding result and the phoneme duration corresponding to each phoneme to obtain a phoneme code. In step S405, the phoneme code is taken as a query for the energy-aware attention mechanism, the energy code is taken as a key and a value for the energy-aware attention mechanism, and the energy-aware attention mechanism is processed to obtain a new energy code. In step S406, the pre-trained song synthesis model further includes a decoder, the phoneme code and the new energy code are fused to obtain a code fusion result, and the code fusion result is processed by the decoder to obtain a synthesized song mel-spectrum. In step S407, a synthesized song audio of the song to be synthesized is generated according to the synthesized song mel-spectrum. It should be noted that the specific definition of the above steps can refer to the specific definition of the song synthesis method described above, and will not be repeated here.

[0105] It should be understood that although each step in the flowchart involved in each embodiment as described above is displayed in sequence according to the arrow, these steps are not necessarily executed in sequence according to the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other orders. Moreover, at least part of the steps in the flowchart involved in each embodiment as described above can include multiple steps or stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence of these steps or stages is not necessarily sequential, but can be executed alternately or alternately with at least part of other steps or steps or stages in other steps.

[0106] Based on the same inventive concept, the embodiments of the present application also provide a song synthesis device for implementing the above-mentioned song synthesis method. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme described in the above method, so the specific definition in one or more song synthesis device embodiments provided below can refer to the definition of the song synthesis method in the above text, and will not be repeated here.

[0107] In one embodiment, as shown in FIG. 1, a singing voice synthesis device is provided, comprising: Figure 5

[0108] A musical score information and energy information obtaining module 501 is configured to obtain musical score information of a to-be-synthesized song and to-be-input energy information; the musical score information comprises a plurality of phonemes and a pitch and a phoneme duration corresponding to each phoneme; the to-be-input energy information is obtained based on the musical score information or extracted based on reference audio of the to-be-synthesized song;

[0109] A singing voice synthesis model processing module 502 is configured to input the musical score information and the to-be-input energy information into a pre-trained singing voice synthesis model, and obtain a synthesized singing voice mel-spectrogram through energy-aware attention mechanism processing in the pre-trained singing voice synthesis model; the energy-aware attention mechanism is configured to combine phoneme encoding of the musical score information and energy encoding of the to-be-input energy information to obtain new energy encoding;

[0110] A synthesized singing voice audio obtaining module 503 is configured to generate synthesized singing voice audio of the to-be-synthesized song according to the synthesized singing voice mel-spectrogram.

[0111] In one embodiment, the singing voice synthesis model processing module 502 comprises:

[0112] A phoneme encoding obtaining sub-module is configured to obtain the phoneme encoding according to the plurality of phonemes and the pitch and the phoneme duration corresponding to each phoneme;

[0113] An energy-aware attention mechanism processing sub-module is configured to obtain the new energy encoding through the energy-aware attention mechanism processing based on the phoneme encoding and energy encoding of the to-be-input energy information;

[0114] A synthesized singing voice mel-spectrogram obtaining sub-module is configured to obtain the synthesized singing voice mel-spectrogram according to a coding fusion result of the phoneme encoding and the new energy encoding.

[0115] In one embodiment, the pre-trained singing voice synthesis model comprises a text encoder, and the phoneme encoding obtaining sub-module comprises:

[0116] A text encoder processing unit is configured to input the plurality of phonemes and the pitch corresponding to each phoneme into the text encoder to obtain a phoneme encoding result;

[0117] A phoneme expansion unit is configured to perform phoneme expansion according to the phoneme encoding result and the phoneme duration corresponding to each phoneme to obtain the phoneme encoding.

[0118] In one embodiment, the energy-aware attention mechanism processing sub-module comprises: ​

[0119] a new energy encoding obtaining unit, configured to obtain the phoneme encoding as a query for the energy-aware attention mechanism, obtain the energy encoding as a key and a value for the energy-aware attention mechanism, and process the energy encoding through the energy-aware attention mechanism to obtain the new energy encoding.

[0120] In an embodiment, the pre-trained song synthesis model further comprises an energy predictor or an energy extractor, and the synthesized song mel-spectrogram obtaining submodule comprises:

[0121] a code fusion unit, configured to fuse the phoneme encoding and the new energy encoding to obtain the code fusion result.

[0122] a decoder processing unit, configured to process the code fusion result through the decoder to obtain the synthesized song mel-spectrogram.

[0123] In an embodiment, the score information and energy information obtaining module 501 comprises:

[0124] a to-be-input energy information obtaining module, configured to obtain, through the energy predictor, energy information predicted based on the phoneme encoding as the to-be-input energy information, or extract, through the energy extractor, energy information of the reference audio as the to-be-input energy information.

[0125] In an embodiment, the pre-trained song synthesis model is obtained through the following method:

[0126] a training data obtaining module, configured to obtain training data; the training data comprises a plurality of sample phonemes in a sample song, sample pitch and sample phoneme duration corresponding to each of the sample phonemes, sample energy information and a sample song mel-spectrogram of the sample song; the sample song mel-spectrogram is used as a sample label corresponding to the sample song;

[0127] a predicted song mel-spectrogram obtaining module, configured to input the plurality of sample phonemes, the sample pitch and the sample phoneme duration corresponding to each of the sample phonemes, and the sample energy information into a song synthesis model to be trained to obtain a predicted song mel-spectrogram.

[0128] a model training module, configured to train the song synthesis model to be trained based on a loss between the predicted song mel-spectrogram and the sample song mel-spectrogram until a preset training end condition is met, to obtain the pre-trained song synthesis model.

[0129] a sample obtaining submodule, configured to obtain song audio data and sample score information of the sample song; the sample score information comprises the plurality of sample phonemes, sample pitch and sample phoneme duration corresponding to each of the sample phonemes.

[0130] extracting a sample energy information and a sample singing mel-spectrogram from the singing audio data of the sample song.

[0131] The modules in the singing synthesis device can be implemented by software, hardware, or a combination thereof. The modules can be embedded in or independent of a processor in a computer device in hardware form, or stored in a memory in the computer device in software form, so as to be invoked and executed by the processor to perform the operations corresponding to the modules.

[0132] In an example embodiment, a computer device, which can be a terminal, has an internal structure as shown in Figure 6 The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit, and an input device. The processor, the memory, and the input / output interface are connected through a system bus, and the communication interface, the display unit, and the input device are connected to the system bus through the input / output interface. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is configured to exchange information between the processor and external devices. The communication interface of the computer device is configured to perform wired or wireless communication with external terminals. The wireless communication can be achieved through WIFI, mobile cellular network, NFC (Near Field Communication), or other technologies. The computer program is executed by the processor to implement a singing synthesis method.

[0133] Those skilled in the art can understand that Figure 6 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not limit the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0134] In an example embodiment, a computer device includes a memory and a processor. The memory stores a computer program. The processor executes the computer program to implement the following steps:

[0135] obtain score information of a song to be synthesized and input energy information; the score information includes a plurality of phonemes and corresponding pitch and phoneme duration of each phoneme; the input energy information is obtained based on the score information or extracted from reference audio of the song to be synthesized;

[0136] inputting the score information and the to-be-input energy information into a pre-trained singing synthesis model, and obtaining a synthesized singing mel-spectrogram through an energy-aware attention mechanism in the pre-trained singing synthesis model; the energy-aware attention mechanism is used to combine phoneme encoding of the score information and energy encoding of the to-be-input energy information to obtain new energy encoding;

[0137] generating a synthesized singing audio of the to-be-synthesized song according to the synthesized singing mel-spectrogram.

[0138] In one embodiment, the processor, when executing the computer program, also implements the steps in the other embodiments described above.

[0139] In one embodiment, a computer-readable storage medium is provided, and the computer-readable storage medium stores a computer program. The computer program, when executed by a processor, implements the following steps:

[0140] obtaining score information of a to-be-synthesized song and to-be-input energy information; the score information includes a plurality of phonemes and corresponding pitch and phoneme duration of each phoneme; the to-be-input energy information is obtained based on the score information or extracted based on reference audio of the to-be-synthesized song;

[0141] inputting the score information and the to-be-input energy information into a pre-trained singing synthesis model, and obtaining a synthesized singing mel-spectrogram through an energy-aware attention mechanism in the pre-trained singing synthesis model; the energy-aware attention mechanism is used to combine phoneme encoding of the score information and energy encoding of the to-be-input energy information to obtain new energy encoding;

[0142] generating a synthesized singing audio of the to-be-synthesized song according to the synthesized singing mel-spectrogram.

[0143] In one embodiment, the computer program, when executed by a processor, also implements the steps in the other embodiments described above.

[0144] In one embodiment, a computer program product is provided, and the computer program product includes a computer program. The computer program, when executed by a processor, implements the following steps:

[0145] obtaining score information of a to-be-synthesized song and to-be-input energy information; the score information includes a plurality of phonemes and corresponding pitch and phoneme duration of each phoneme; the to-be-input energy information is obtained based on the score information or extracted based on reference audio of the to-be-synthesized song;

[0146] input the score information and the to-be-input energy information into a pre-trained song synthesis model, and obtain a synthesized song mel-spectrogram through an energy-aware attention mechanism in the pre-trained song synthesis model; the energy-aware attention mechanism is used to combine a phoneme code of the score information and an energy code of the to-be-input energy information to obtain a new energy code;

[0147] generate a synthesized song audio of the to-be-synthesized song according to the synthesized song mel-spectrogram.

[0148] In one embodiment, the computer program, when executed by the processor, also implements the steps in the other embodiments described above.

[0149] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0150] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., without being limited thereto.

[0151] Any combination of the technical features of the above embodiments can be made. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described, however, as long as the combination of the technical features does not exist, it should be considered as the scope of the present application.

[0152] The above embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent of the present application. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of protection of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A method for synthesizing singing voice, characterized in that, The method includes: The musical score information of the song to be synthesized and the energy information to be input are obtained; the musical score information includes multiple phonemes and the pitch and duration of each phoneme; the energy information to be input is predicted based on the musical score information or extracted based on the reference audio of the song to be synthesized. The musical score information and the energy information to be input are input into a pre-trained vocal synthesis model. The energy-aware attention mechanism in the pre-trained vocal synthesis model is used to process the synthesized vocal score and obtain a new energy code. The synthesized vocal score is obtained by combining the phoneme encoding of the musical score information and the energy encoding of the energy information to be input. Based on the synthesized vocal spectrum, the synthesized vocal audio of the song to be synthesized is generated.

2. The method according to claim 1, characterized in that, The step of inputting the musical score information and the energy information to be input into a pre-trained singing synthesis model, and processing them through the energy-aware attention mechanism in the pre-trained singing synthesis model to obtain the synthesized singing Mel spectrum, includes: The phoneme encoding is obtained based on the multiple phonemes and the pitch and duration of each phoneme. Based on the phoneme encoding and the energy encoding of the energy information to be input, the new energy encoding is obtained through the energy perception attention mechanism. The synthesized vocal spectrum is obtained based on the encoding fusion result of the phoneme encoding and the new energy encoding.

3. The method according to claim 2, characterized in that, The pre-trained singing synthesis model includes a text encoder, wherein obtaining the phoneme encoding based on the multiple phonemes and the pitch and duration corresponding to each phoneme includes: The multiple phonemes and the pitch corresponding to each phoneme are input into the text encoder to obtain the phoneme encoding result; Based on the phoneme encoding result and the phoneme duration corresponding to each phoneme, phoneme expansion is performed to obtain the phoneme encoding.

4. The method according to claim 2, characterized in that, The energy encoding based on the phoneme encoding and the energy information to be input, processed by the energy-aware attention mechanism to obtain the new energy encoding, includes: The phoneme encoding is used as a query for the energy-aware attention mechanism, and the energy encoding is used as a key and value for the energy-aware attention mechanism. After processing by the energy-aware attention mechanism, the new energy encoding is obtained.

5. The method according to claim 2, characterized in that, The pre-trained singing synthesis model further includes a decoder, wherein the synthesis singing Mel spectrum is obtained based on the encoding fusion result of the phoneme encoding and the new energy encoding, including: The phoneme encoding and the new energy encoding are fused to obtain the encoding fusion result; The decoder processes the encoded fusion result to obtain the synthesized vocal mel spectrum.

6. The method according to claim 1, characterized in that, The pre-trained singing synthesis model further includes an energy predictor or an energy extractor, wherein obtaining the input energy information includes: The energy predictor uses the energy information predicted based on the phoneme encoding as the input energy information. Alternatively, the energy information of the reference audio can be extracted using the energy extractor and used as the input energy information.

7. The method according to any one of claims 1 to 6, characterized in that, The pre-trained singing voice synthesis model was trained using the following method: Acquire training data; the training data includes multiple sample phonemes in the sample song, as well as the sample pitch and duration of each sample phoneme, the sample energy information of the sample song, and the sample vocal melodic spectrum; the sample vocal melodic spectrum is used as the sample label corresponding to the sample song; The sample phonemes, the corresponding sample pitch and duration of each sample phoneme, and the sample energy information are input into the singing synthesis model to be trained to obtain the predicted singing Mel spectrum. Based on the loss between the predicted singing Mel spectrum and the sample singing Mel spectrum, the singing synthesis model to be trained is trained until the preset training termination condition is met, thus obtaining the pre-trained singing synthesis model.

8. The method according to claim 7, characterized in that, The acquisition of training data includes: The audio data of the sample song and the sample score information are obtained; the sample score information includes the multiple sample phonemes and the sample pitch and sample phoneme duration corresponding to each sample phoneme. Based on the audio data of the sample songs, the energy information and the Mel spectrum of the sample songs are extracted.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 8.

Images